{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discrete and Continuous Control with A2C and PPO\n",
    "\n",
    "Reinforcement learning (RL) is the study of agents taking actions in an environment in order to increase their \"reward\". This takes place in a perception-action-learning loop, whereby the agent receives a state/observation at every timestep, uses its policy to choose an action conditional on this observation, and then receives another observation and a scalar reward as the environment transitions to the next timestep. Through multiple interactions with the environment, the goal of the agent is to improve its policy so that it can maximise its expected cumulative reward.\n",
    "\n",
    "![RL Loop](https://raw.githubusercontent.com/torch/torch.github.io/master/blog/_posts/images/action-perception.png)\n",
    "\n",
    "Unlike the usual supervised or unsupervised learning settings, the agent is responsible for the data it receives, and so it is far from \"independent and identically distributed\". Furthermore, there is a particularly difficult credit assignment problem - the agent has to work out which of its actions result in rewards, where the rewards caused by specific actions may appear a long time in the future. Nevertheless, RL provides a formal framework for studying this problem. In contrast to the field of optimal control, we're not assuming that we have access to a (dynamics/forward) model of the environment. RL also includes the concept of model-based methods, which attempt to learn and use such models from data, but that won't be covered here.\n",
    "\n",
    "Here we'll look at the advantage actor-critic (A2C) algorithm, which combines an explicit policy (the actor) with value function estimation (the critic); the value function captures the expected cumulative reward from a given state and given a certain policy, or more informally how good a state is. We'll also look at a more sophisticated algorithm - proximal policy optimisation (PPO) - that aims to prevent disastrously large updates to the policy.\n",
    "\n",
    "We'll look at learning to solve two classic control problems - cartpole and pendulum - using A2C. A2C has difficulty in learning to control the pendulum, so we'll show how PPO's more conservative updates actually help solve this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environments\n",
    "\n",
    "Firstly we'll instantiate the two environments and see how a random policy performs over several rollouts. This'll give us an idea of the problem to solve, as well as how a naive baseline would perform on it. These are both episodic environments, which means that they can terminate at some point. With cartpole, a random policy fails to keep the pole balanced and episodes terminate quickly. With pendulum, a random policy is unlikely to swing the pole up into the upright position, so episodes time out with low rewards.\n",
    "\n",
    "Just to note - we'll look at renders of the environment, but the observations for the agents will be a few symbolic inputs, such as the angle of the pole in the cartpole environment. However, the ability to learn policies \"end-to-end\", i.e., directly from pixels, is one of the major success of deep reinforcement learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import gym\n",
    "from matplotlib import pyplot as plt\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from torch.distributions import Categorical, Normal\n",
    "from torch.nn import functional as F\n",
    "from IPython.display import clear_output, display\n",
    "%matplotlib inline\n",
    "gym.logger.set_level(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Average reward: 14.60'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAABHNJREFUeJzt29FNG0EUQNHdyFXQRtIGbUBNuI20kbSRNjY/VmSBIcgW3t0750hIGAn0PszVaOd5XpZlAqDn29oDAPA1BB4gSuABogQeIErgAaIEHiBK4AGiBB4gSuABog5rD3Di47QAb823/LITPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRB3WHgC26vfx+c3Pvj+9rDAJXMcJHiBK4AGiBB4uuPR4BvZG4AGiBB4gSuDhk2zQsDcCDxAl8ABRAg+v+IATFQIPECXwcMb+OyUCDxAl8ABRAg8QJfDwHzZo2CuBB4gSeDixQUONwANECTxAlMADRAk8fMAGDXsm8ABRAg+TDRqaBB4gSuABogQe3uGClb0TeIAogWd4LlipEniAKIEHiBJ4uMAFKwUCDxAl8ABRAs/QbNBQJvAAUQIPECXw8IoNGioEHiBK4BmWC1bqBB7OeDxDicADRAk8QJTAA0QJPECUwDMkGzSMQODhxAYNNQIPECXwDMfjGUYh8ABRAg8QJfAwuWClSeABogSeobhgZSQCDxAl8ABRAg8QJfAMzwYNVQIPECXwDMMGDaMReHZtnudPf936N2BvBB4g6rD2AHBPP/88/fv+8eE4/Xg+rjgNfC0neIZxHvdLr6FG4BnCrxcxZzwCDxAl8Azj8eH44WuomZdlWXuGaZqmTQzB/txzfXEj/yuM5aY3+Ca2aOwYswfep9zbrYeKTQTeyYhrOcHD+zyDB4gSeIAogQeIEniAKIEHiBJ4gCiBB4jaxB48XMtuOrzPCR4gSuABogQeIErgAaIEHiBK4AGiBB4gSuABogQeIErgAaIEHiBK4AGiBB4gSuABogQeIErgAaIEHiBK4AGiBB4gSuABogQeIErgAaIEHiBK4AGiBB4gSuABogQeIErgAaIEHiBK4AGiBB4gSuABogQeIErgAaIEHiBK4AGiBB4gSuABogQeIErgAaIEHiBK4AGiBB4gSuABogQeIOqw9gAn89oDANQ4wQNECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0T9BaadSwpzPRCMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.axis('off')\n",
    "\n",
    "env = gym.make('CartPole-v0')\n",
    "env.reset()\n",
    "rollouts = 5\n",
    "total_reward = 0\n",
    "view = plt.imshow(env.render(mode='rgb_array'))\n",
    "\n",
    "# Perform several rollouts/episodes with a random policy\n",
    "for _ in range(rollouts):\n",
    "    _, done = env.reset(), False  # Reset the environment to a starting state\n",
    "    while not done:  # Act until episode terminates\n",
    "        view.set_data(env.render(mode='rgb_array'))\n",
    "        display(plt.gcf())\n",
    "        clear_output(wait=True)\n",
    "        _, reward, done, _ = env.step(env.action_space.sample())  # Sample random (valid) action\n",
    "        total_reward += reward\n",
    "\n",
    "env.close()\n",
    "display('Average reward: %.2f' % (total_reward / rollouts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Average reward: -1299.98'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQsAAAD8CAYAAABgtYFHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAA/BJREFUeJzt2jFuWkEUQNFPRJndJJ3lHZgtpMjCUmQL9g4sd/FuUloiRSzkxAjfBPBn4JyKQfrSk0BXwzCL9Xo9Abzlw9wDAGMQCyARCyARCyARCyARCyARCyARCyARCyBZzj3AM9dI4fgW+zxsZwEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEky7kHYByPq9Wr9z7d3s4wCXNYrNfruWeYpmk6iSH4bVsUdhGMYSz2elgsLs+/xmCbrw8P07erq81aMIYgFmx3iCj87fPd3R/rHzc3m9eCcfL2ioUzizNxjDAUP5+epo9LX6NL4FMewFwhgJfE4oSMGAW7isvhzGImI4bhpS/399P36+vN2nnFEBxwjmD0OOwiFMPYKxZucL4DoeAc+MF5ZOcUCmG4bGLBK6LANmJxgcSA/yEWZ0wUOCSxOBPCwLH56/Qd7HvIKQQciHsWIyjBEAWOTCxG87haCQNzEAsgcYMTOD6xABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxABKxAJLl3AM8W8w9ALCbnQWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQiAWQ/AIhEGGVq+c/MQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.axis('off')\n",
    "\n",
    "env = gym.make('Pendulum-v0')\n",
    "env.reset()\n",
    "total_reward = 0\n",
    "view = plt.imshow(env.render(mode='rgb_array'))\n",
    "\n",
    "for _ in range(rollouts):\n",
    "    _, done = env.reset(), False\n",
    "    while not done:\n",
    "        view.set_data(env.render(mode='rgb_array'))\n",
    "        display(plt.gcf())\n",
    "        clear_output(wait=True)\n",
    "        _, reward, done, _ = env.step(env.action_space.sample())\n",
    "        total_reward += reward\n",
    "\n",
    "env.close()\n",
    "display('Average reward: %.2f' % (total_reward / rollouts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "We'll construct two small fully-connected networks (an actor network and critic network) with one hidden layer each for both environments. For discrete control in cartpole (left or right) we'll use a categorical policy, and for continuous control in pendulum (torque) we'll use a Gaussian policy. We'll make the standard deviation of the Gaussian policy independent of the input, which will keep it more stable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Agent(nn.Module):\n",
    "    def __init__(self, obs_size, env):\n",
    "        super().__init__()\n",
    "        self.env = env\n",
    "        # Create separate actor and critic networks in one module (they do not directly interact)\n",
    "        self.value_fc1 = nn.Linear(obs_size, 256)\n",
    "        self.value_fc2 = nn.Linear(256, 1)\n",
    "        self.policy_fc1 = nn.Linear(obs_size, 256)\n",
    "        self.policy_fc2 = nn.Linear(256, 1 if env == 'pendulum' else 2)\n",
    "        if env == 'pendulum':\n",
    "            self.std_dev = nn.Parameter(torch.zeros(1))  # Make the standard deviation a standalone parameter\n",
    "\n",
    "    def forward(self, obs):\n",
    "        value = self.value_fc2(torch.tanh(self.value_fc1(obs)))[:, 0]\n",
    "        policy = self.policy_fc2(torch.tanh(self.policy_fc1(obs)))\n",
    "        if self.env == 'cartpole':\n",
    "            policy = Categorical(logits=policy)\n",
    "        elif self.env == 'pendulum':\n",
    "            policy = Normal(policy[:, 0], F.softplus(self.std_dev))\n",
    "        return policy, value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A2C\n",
    "\n",
    "### Training and Testing\n",
    "\n",
    "The first part of A2C training involves taking several rollouts in the environment. The environment is reset to start a new episode, and the agent takes actions sampled from its policy until the episode terminates. Meanwhile, we'll need to keep track of the policy, value estimates and rewards encountered at every timestep.\n",
    "\n",
    "Training here utilises a Monte Carlo approach to estimating the expected return (the expected sum of discounted rewards, where a discount factor $\\in (0, 1)$ introduces a myopic bias that reduces variance). The value function is trained to estimate the return at each time step, and this can be treated as a regression problem in which the L2 norm between the value function estimate and the actual return is to be minimised. The residual - the difference between the return and the value function estimate - is known as the advantage, or how much better the policy did than expected. Technically the original A2C algorithm uses n-step returns, but we'll instead use a lower variance estimation of the advantage, known as generalised advantage estimation (GAE), to train the policy. GAE utilises temporal difference (TD) errors and eligibility traces in order to average between different n-step returns at once.\n",
    "\n",
    "The policy is trained using the REINFORCE rule, also known as the score function estimator or log derivative trick. Concretely, the parameters of the policy are updated to increase the log-likelihood of the taken action, weighted by the return. However, the return can have a high variance, so the advantage is utilised instead. It can be shown that using the return minus a baseline (in this case, the estimated value) can reduce the variance, and does not introduce bias if the baseline is independent of the action (in practice, there may be another sort of bias from the value function estimator being inaccurate though). Importantly, we have to use `.detach()` on the `advantage` to make sure that the parameters of the value function estimator are not influenced by the policy gradient loss. Also note that we \"subtract\" the loss, as we want to perform gradient *ascent* on this, not *descent*.\n",
    "\n",
    "A2C also uses entropy regularisation, whereby more entropic policies are promoted. This prevents the policy from collapsing into a deterministic solution too soon. Note that this isn't quite the \"maximum entropy\" principle from RL, as that also requires taking into account the entropy of the policy in future states.\n",
    "\n",
    "Finally, we'll examine the final policy learned for each environment. If it has learned well then it should be significantly better than the random policies we observed earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Average final reward: 167.56'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0cAAAF3CAYAAABuXzAHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm8HXV9//H3J7nZCIQAuUAEYgAjsge8LJWiLMpWJWBFoVbDJkUBgWol2Nati4qCCggWympZ+0MWW7YYEQoS4UZoAEFZDBLIxhoCSUhuPr8/Zk7PnHNnzj4zZ3k9H4/7mJnvbJ8799xz5nO+3/l+zd0FAAAAAL1uRN4BAAAAAEA7IDkCAAAAAJEcAQAAAIAkkiMAAAAAkERyBAAAAACSSI4AAAAAQBLJEQAAAABIIjkCAAAAAEkkRwAAAAAgieQIAAAAACRJfXkH0IxJkyb51KlT8w4DAAAAQBubN2/ey+7eX227jk6Opk6dqsHBwbzDAAAAANDGzOz5WrajWR0AAAAAiOQIAAAAACSRHAEAAACApBSTIzPbyszuMbMnzewJMzs9LN/YzGab2dPhdKOw3MzsfDN7xszmm9nuacUGAAAAAOXSrDlaK+lL7r69pL0lnWJmO0iaJWmOu0+TNCdclqRDJU0Lf06SdHGKsQEAAABAidSSI3df5O6/DefflPSkpC0kzZB0VbjZVZKOCOdnSLraA3MlTTSzyWnFBwAAAABRmTxzZGZTJe0m6TeSNnP3RVKQQEnaNNxsC0kvRHZbGJaVH+skMxs0s8Fly5alGTYAAACAHpJ6cmRm60u6SdIZ7r680qYxZT6swP0Sdx9w94H+/qrjOAEAAABATVJNjsxslILE6Bp3/1lYvKTQXC6cLg3LF0raKrL7lpJeSjM+AAAAAChIs7c6k3SZpCfd/bzIqtskzQznZ0q6NVL+2bDXur0lvVFofgcAAAAAaetL8dj7SPqMpMfM7NGw7KuSviPpRjM7QdKfJB0Vrrtd0mGSnpH0tqTjUowNAAAAAEqklhy5+/2Kf45Ikg6M2d4lnZJWPAAApMbCj7t33pFGjco3ljirV0tjxmR3PndpaEjqq+E2Y2hIGjkyft3ixdKmm0ojGLMeQDZ4twEAoFVGj847guHMpLFjiwlc2u64I0hmRo2S1q6tvO2uuwYJVFxss2dLkycnJ06VmAU/Pqxfp/gyAAiRHAEAgNa44w7psMOKy3//95W3nz+/OL/ttqXrDjqoOB+XIK1ZE3/MaKI1Zcrw9SNGBNuMH185NgA9ieQIAIBudfPN6R5/3brifHliJEnnnJO8b3lt0XPPFed/8Yvk8xSMHh0cY9y45HMsXJi87u23k9cB6FlpdsgAAED+Cjfhy5dLG2yQbyxZ+/jHS5eTmprVK60mepMnS4sWSR/5SO3nX7Wq+Htl1XQQQNei5ggAelnh2Ywsbypbcb7CjXA9x5kwoblzJummG/KhoeDafulLpa+N8p9Kbr9d+vM/r7xN9BjRZG3x4tJao+nT4/epdszjj6++TS3HBNBzSI4AAIFNNkn/HNGb0VpudpNuxqO9l1Xarpdvfv/iL4rztdaY9fUF1/a886pvKwWJzX//d3H5+uulQw+V/ud/kvfZZZfifKFJ3N13F8uitUaPPDJ8/6TEKuqyy4rz996bHAsAlCE5AgAEXn219pviVml18lJP7UKcpUvjn2+pxzXXNLd/q9x+e3F++fLWHtu9mJgcdlhx+VOfGr7tzjuXLj/2WHG+8NxPXDO6QgcN0QRozpz4WCot77ff8H2inTF8+9vD1xdkXasKIHfmHdyl5cDAgA8ODuYdBgB0pqSbvlZ8LixdGoxP0+g5y7eLrt9ww+LNfqE8uv1ee0lz5zb2+xX2qecaxJ2nHT5by2tYqtW41FIj08z5y8vriaGWa/zWW6VJT9yxoueu5fctbPPAA9IHPhC/DYCOYGbz3H2g2nbUHAEApI99rDjf7DflZtJmm1V+vqNaMhR3zIK4WpDo8X7zm8rrL7+8+jk6sbZgyZLi/GabFefbIVErqPW69vfXf+zyrrn32qt0eerU0uWxYysfL1rjtM8+9ccDoCORHAEApNtuK102C54/qccFF9SXVCQlSLUeo3y7Sy+NX1d+nhNOqO34ncRM2nzzYjOwpUuHb1OpW+/DDy/OtyqZuuOO+vcpNM8rj79a07k4c+eWLj//fOnyypXF+bjXOs8qIW/nnVf8n+7rky68MO+IegLJEQD0usKNpntppwxDQ7X3VGYmffGL8eXlPvGJ4eeuFpsk7bFH6fHKnw068cTKx0pbredv9XMslY4VHVj1iCOK83vsUbrdz3/eungKDjkkeV3WtVkbbVR5/dBQ9WMccEBrYgFq9aUvFeeHhqTTTiu+f+y2W35xdTmSIwBA0csvV79x/dGPKq8///zhZdEb+P/8z9J1Sc+WXHxx6Xa1PGNaHvvee1c/T9yyJF11VfXzRfeL1lzVohUJUvQYI2I+0p95Jn6/pGs5ZkzzMcUZMaL533f16mBaSyJT7vXXi/PR18F999V+jHvuKV0u3KT+4Af1xwNUM3p0cf5znxv+//Poo/FfXpUPoIy60SEDAHSbdeukNWsq3+jW8zB6nKREo9pD9Ennq/TAfdy6f/s36aST4o9f67NNjcZa6XjVOnQoP88hhzTW/CzuWHFNz2p57qvVHTEkna88tizU0pFD3O//wAPFsZr237+YGFV6TXbw/RQy1tcXvD+/9Vb8+osukk45JZj/ylek7363dP3220tPPZV8fDNpwQJpypSWhNst6JABADpRvQNuxhk5MnjYvBUDrUZ/0lR+/Pe/P3mdlJwYFbb/05/i96uWEM2cWTnOWo0cWdt2d97Z2PHLa4nqTTgL62+9tbHzN2PWrOzO1ejrNjqI7S9/WZw3kx58MH6fwnNfQCWTJgW1n2+/nfweXUiMzIYnRpL05JPF9+V164LpLbcUvxBzl9797mzGrutCJEcA0C4qPc/T6mM2InqjWeimO9qErpaH5ivdrEbXNdsqYKutatvuiitKr9GVV7YmEaxnrKSkv9GNNyYnyNEYm4k3+hzSmWc2fpwkcbFVGlcoD/XGE+3Su/z3i/YYCJR77TXplVdKy8yCJnJSMAZY9IuPWt5HCu8PM2ZIq1ZJy5YVv5x59dXkprVIRHIEAO0gza6jKx37+99v7JjLlgXT009vbP8kSbVUrUoGyvc//vjK29b6d0mjZi1uQNVy06fXf1z34R0ySNkPAJyHuB7oojVZ73pX6br77w+mDzwwfL8zzgim5a/ZTuwGHtnYeOPi/HXXFed32y143eyyS/G19PnPN3aOSZOktWuLtUjve19jx+lhJEcAkLe4Z0cabcpWb+1TtDekNMR1ztAOor24FWQ5lk29N9RJYzA98khj53/ooeD8EyY0tn+jdt452/OV++AHK69ftKh0ufCaiBsAlo4YEDVqVPC/mdSRyw47FOeffVY6+ujk9/g99gieO2rGa68F06Ehao/qRHIEAHmq9lB90na1SGrmFv32splj1vIg/2mnFeej48o0eu5W1dDE3SwUagkk6dhjGztuM/HdcENxPqvahzfeyOaZsoL587M5T1QtXaxvvXVxvpZrX612c/vtqx8D3WXt2mB67LHSFlsMX//kk8F0gw2kbbYplhfG9Yp+KfbQQ83HM24ctUcNIjkCgLxMnVq63Mob1OjYRQWFtuyFbxSzUvjAHzs22/NWU+l6X3FFcT6pg4y4rrObieHooytvW94FeuGhbVR26aXBw++V/t7PPVf9OO5Bgl/L/2mlnsTQfcp7Bn3ppeC9Ytq04e8by5cP37+/P524qD1qCMkRAOTl+eeL80k3XP/6r7Ufr5ZvvFtRIxF9iF+qPu5RJ/jxj2vftlp33QXXXBO/X7n99ivdJqlG7pOfLF134YVVQ20L7sHNWZ5dXY8bV/8+0W/3C6ol+HPnFufbtUkpKmukl9B33gmm22xTmiiVJySXXdZcbPWKqz266KLgdbz++vGJGhjnCAByU+v4Mq3YLu7D3qy+XtUaiamTjR1bHHi0XPkYQvWO+1TL3yfuPOXr0Dq1NnGt5zj8jTpP4e/3pS/V1mHNaacVv6go/L2/8IXiINajRkmLFzfXnLkZK1dK662XvH7CBOmFF7J//jAHjHMEAO0szRuouHEx4s7RaGLUK1atSh7rqdXPBFV6hiVuXXkTOzSvVf+HBx/cmuMge4UhCiTp3HNr2yeuBveii4rvGe+8k19iJAW1R+U1p1OnBjVHUlB7tOGG0gEHZB5auyI5AoB2V0vTuujN+le+kl4sBVkNDtsJfvrT+rbv62v+nJ/4RPPHQDqiA/vSrXdnKQxRUDBQtZIhed928tZb0je/WXzP/uMfpTffDDqCKCRJ99wj3XVXvnG2CZrVAUDWGqk1qrZPq5voIVm15lf77y/96lfD19XyrFJhm9WrpdGjk8/L3y4dm28eDOT6V381/JmxevC36kzVBlwuN2JE5RreTrHTTtITTwTzce89XYJmdQDQi6ZNq7x+u+2yiaOX3XNP4/sWvtmtdHNy6aWNHx+VLV4cXP9mEiOp9Eb5sMOaOxayF/37TZxYfbtopyqd6PHHg2ejpJ549qiaFtTtAwAa8u53177t+PFB0wgp+HYz6fmXP/yh8nHoYrh5lTpJKFf4W113XWvOi85zxx15R4BaLF5cujxmTFCL8sYb8dtHn9Fp5guRdrFiRfF3HhiQerhlFjVHAJCl6E31ggW177diRfxxeKahM/zVX+UdAbJ26KHJ684+O7s4UJvyrttXrSrOxz0n2A0JUdTo0dKNNwbz8+b1dDffJEcA0Cmqdf9MzUJ2qo3dw98Ct99enI/+r5pJ3/kOX2y0m5Urh5dtuWUwHRpK3m/p0nTiycNRRxWb9LZzBxMpIzkCgKz80z8V5xu9eeZmvH2MqPEjlJtgFFR7LRx/fGODkKJ1Tj+9OP/CC8X5D3+4OF94PkeS+vvTjylLhWesrrgi3zhyRHIEAFn52tdac5xvf7t0mcSoc+y4Y94RIEv1jo3VwzekbeOHP4wvnzOnOL92bTDdYov048naVlsF0/vuyzeOHJEcAUCnmTUr6KBBIjFqZ3E9nj3+ePZxoH1V6l3y3/89uzh6XXlnDFHR99jHH5duvbW4vHBhejHlZe+9g+mzz+YbR45IjgC0j26+0Y/2TNeK33PFiu6+Xt2AThgQJ/p/+8wzxfl//MfS7T73uWziQe01QDvvLB1xRLqx5G3mzGD62mv5xpEjkiMA7WPEiGJ7++hPN/jTn/KOAEAeoslQYX799Ydv98//nE08GG7dusrrv/Wt4WUXXJBOLHnbY49gunp1vnHkKLXkyMwuN7OlZvZ4pOwGM3s0/FlgZo+G5VPNbGVk3U/SigsAgMzsv3/eEaAdFAb3LXjzzeRtx4wpztfa6QdaI2k8svJaPUk69dR0Y8lbtYSxi6U5COyVki6UdHWhwN0/VZg3s3MlRUfWetbdp6cYD4BOUT7IZvmgp50m+rt08u+B+v3yl91T+4l0mJUOuLlqVfE1w/tFOpKu79FHJ+8zaZL08svBPElrV0vtr+vu90l6NW6dmZmkT0pqwZDhALoCY/YA6FUDA6XL8+blE0e3OOus5GbZW29dnDeTPvSh2o4ZHfdnzZrm4mt3PZ785fXb7ytpibs/HSnb2sweMbN7zWzfnOIC0E6+/OW8I2jelCnF+WhzGfSONWukd94h4Uepiy9OXrf77sV5ah7rd845yesWLChdrqfL6iVLgpq9bk8eCp9Vjz6abxw5yeuve4xKa40WSZri7rtJ+ltJ15rZhLgdzewkMxs0s8FlPTx6L9BW0uo84XvfKz1HpzErHURw1ar8YkF++vpKB40EJOnkk4eXkUC33osvxpf/4Aely7V8xmy6aW98ybXRRsG0R8fdyjw5MrM+SR+XdEOhzN1Xu/sr4fw8Sc9Kem/c/u5+ibsPuPtAf7eNSgygsZuDduzZjmaCAJoRfc/4zW/yi6PTbbllfPkZZwTXeK+9guWhoexianeFpocPPphvHDnJo+bow5Kecvf/GznLzPrNbGQ4v42kaZKeyyE2AEnWrcsn+ejEpILECEArFQbmRHPGjRteNnfu8E6Aet2+4dMtzz9fWr54sTRnTtd/pqXZlfd1kh6UtJ2ZLTSzE8JVR2t4RwwflDTfzP5X0v+TdLK7x3bmACAnI0cG02ofIJXWR2t4zjyztn2qHbvdP9C6/EMEQBOi7w/RHuuQDpo21+bEE4Pp8uWl5R/4gPThD0uzZ2cfU4bS7K3uGHef7O6j3H1Ld78sLD/W3X9Stu1N7r6ju+/q7ru7+8/TigtAxOc/X1tztGYTkLhz/PCHzR2z0rnaBYkRgFq9//3Dy77xjczD6Erl4xQ9/HA+cXSKbbcNpuUDwRYGM7/00mzjyZh5B394DwwM+CDftACNq3X8nUrNxOKSkWrry7erJY64cSmSjp33+xpjlABoFd5P6nPKKdJFFw0vf/FFaYstgnmuZXWVPnM33TToua/DmNk8dx+otl2X90UIoGZf+EJ8eStrYspHiW/k2ZxCLdRZZ9W/bxa++c28IwDQjTbYIO8IOkM0MZo+vThfSIzQmGhtW5f3Fk1yBCBQacyNcqedVnl9Iempp0aoXtFxLMqPnWfTOprBAEjDihV5R9B5HnlkeFm3j1HUKoXnjAv+9V+DqVnwmdvFYyDxCgF6VS0JRFJyc+GFzZ27kVoe9/g2+dWcfXb7dfMNALUqv0lFbSZNii+Pjj2HZGPHBtNf/jKYPvBAMP3wh4Ppv/xL9jFlhOQIQGu4S1/8Yvy6XXdtzTkGB4tN8wYGiuet5Dvfac25ASAPa9fmHUFniqs1kqR3vSvbODrVJpsE02uvDaavhp1IX311ML3vvuxjygjJEYCiaO1KrZ01RP3oR/H7x1W/lx+z3tqkhx+ufIxZs4bXFmVZe7TfftmdC0BvuPzyvCPoHIXBXxlAtzHTpgXTQsdnQ0NBk8TNN5f6+rr6uSOSI6AXff7zxfl6kpJ2eranmu9+N9/z33NPvucH0H1OOKH6Nii1557F+fXXzy+OTlNoPrdwYbGWaOLEYLrlltWfO/ryl5Nbk7Q5kiOgF/3kJ6XL229fulxLrVG1Thmq7V9tXStFz/PVr2ZzTgBAtqId9cRZvDibOLpBYSDYN98sftm4xx7B9PDDg2nSc0fu0nXXdez1JjkCulG9HRD87nel+9YiqVOGehOe8u69m1V+rL33Ll3+9rcbP65Z8kO+UnvXpAHoXJddVt/2hc+AXuuZrXyIh4LC58z48dnG08kKn3Vr1kgPPRTMf+UrwfTss4Np0nNHv/+99NJLxdqnDtNj/zVAj6mWJFVLSmpNWnbeOb583bra9k/Tgw+25jiFm4xXXmnN8QCgVscf39h+7TL+GzqXe7EzhgMOCKbVnjuaMyeYHnhg+vGlgOQI6AWFJKmvL53ajfnzS5cL39LlXZMSvTGIzm+6aXPHzfv3AtC7+vrq2/6MM9KJo52NG5d3BN2h8Fm3bt3wLuUrPXf0i19IU6dK22yTeohpIDkCuk2lG/ehoeR1tfYe1wnfRFZrqtfFvewA6HKV3sfjRHsR7RV/+EPeEXSHaEK08cal6z72sWBa/tzR0FDQIdGBB3bsF4kkR0A3a/XzPL2g1uvVTHNFAKgXPa3VrtCNN5oTrYErf3630LnRr35VWj5vnvTGGx37vJFEcgT0hkKSlNTMrGDlymDAwW69uY/+XnHJTeEB5vJ1DBoIIG9vvlmcr/TlzAYbDC/r0G/wkbNoE/RZs0rXbb55kLC//HLQ3XdB4XmjwvNJHYjkCOg1lWqTxo4d3q64myUNeltu0aLifFKC9c1vti4uAIgTHaMu6T1rxYri/Jgx6cbTTubOzTuC7hMd5uMDHxi+/sc/DqYnn1wsmzNH2mWX5p/tzRHJEdCtCqNbp63W8Y7aRXkPevV2e57kG99o/hgAUMlFF9W3/apVxflurz364AfzjqD7HHZYME3qEv6zn5U22ki64w5p+fKg9cn993dsL3UFJEdAt0rzgdRo7cn556d3njSY1dZsMO5GYt99g2m15nkAkJZa33+OOSb9WNrJmjV5R9B9Zs4MppMnJ2/zd38XfOl4+unSr38trV5NcgQAHSkuQaqWNCUNeEeCBCBLRx1VnE96tuPaa4Npr32ZU29X50i23npB73PRZ4rKnXVW0HHDtddKd90VXP8Or8UjOQLQmG7oCa9aBxVS8s1E0vb0kgQgbTfeWJy/55769v3Qh4aXRTt76HSFpBCtkdSkLrr+xBOld96RLrhA2muv+E5BOgjJEdBNeuFbwVarpRe/SvuWe+GF5mMCgEZ88Yvx5dH3qmgNeOGZywkTgp5Ku0G0Vg3ZOOecoMZo1aqOb1InkRwBQLJakk2acADIQ9yXMxdcUNv2cR3RdPJgsWedlXcEvW3sWOkv/zKY32yzfGNpAZIjAKjFP/5jfDkPAQPI26671rbdwEDyui9/uTWx5OGcc/KOAJddFgxp8bnP5R1J00iOAKDcP/zD8LJvfSt5+8I3skkJFACkaf780uVRo+K3e/jh0uUbbkgnnrzssEPeEfSu8eOlr30t+bXXQcw7+IHqgYEBHxwczDsMoHGrVwfV0R//uHTTTc0fr9BMooP/r9tGeZMTrimAdjNuXHEsI/faPwPefjvoiUwqfa/r1Pc5PvtQAzOb5+4Vqk8D1BwBeRo7Npj+7Gf5xgEA6DwrVza2XyExkqTzzmtNLHnZZJO8I0CXITkCAADodI0+CH/mma2NI2uvvpp3BOgyJEdAt6g2FgHq02j33gCQh6VL844gX9dfn3cE6BLcTQHt6rjj4rtbTcINfHq4tgDa1UYbte5Yv/td646Vhei4cp/6VH5xoKuQHAHt6sor844AhQFiGVwXQLtqZbOyHXds3bGyMGVK3hGgC5EcAQAAdIuf/jTvCLJHs3K0EK8moBNsu23eEQAAOsFf/3XeEaRn7dpgCIxyQ0PZx4KuRXIE5KWeplrPPVf7tjwfAwC95T3vaW7/Aw5oTRxp6u8PBhgdO7a+53GBOvXlHQCAkBmJDQCgfk8/3VyyMGdOeycb7Rwbuk5qNUdmdrmZLTWzxyNl3zCzF83s0fDnsMi6s83sGTP7vZkdnFZcAAAAXafQgUw3iesQp/B7brppsPzYY9nHha6WZs3RlZIulHR1WfkP3P370QIz20HS0ZJ2lPQuSb8ws/e6O41IgVrwrRoAIG2LFweDzWb1mRPtaMFMWreuuLxkSTYxoOekVnPk7vdJqrV/yRmSrnf31e7+R0nPSNozrdgAAAAQY8KE+PKnnpImT86uZ7jtty/O7713aWIEpCiPDhlONbP5YbO7wshlW0iKjOSlhWEZgAJqhwAAaXvzzfjyaLKShaeeKs4/+GC250ZPyzo5uljStpKmS1ok6dywPO6uL7bhrJmdZGaDZja4bNmydKIEAABAvNGj0z1+9MvAbnuOCm0v0+TI3Ze4+5C7r5N0qYpN5xZK2iqy6ZaSXko4xiXuPuDuA/39/ekGDGRh6tThZfXUEvEhAgBo1uab177tmjXpxfGrXxXnN944vfMACTJNjsxscmTxSEmFnuxuk3S0mY0xs60lTZP0UJaxAZmKJjR//GPlbUl4AABpW7Qo7wgC++9fnH/llfziQM9Krbc6M7tO0n6SJpnZQklfl7SfmU1X0GRugaS/kSR3f8LMbpT0O0lrJZ1CT3VADag1AgCkbcqU6tsUPo+a+Sw699zifLUvDoGUmHfwDdXAwIAPDg7mHQZQv/KkpvxDJWl9dJu47QAAaFRSgpPU1Luw3a67SvPnB/Of/ax01VXNnT8uBqBJZjbP3QeqbZdHb3UAWoEPEQBAGnbdNb58/Pj48kJiJElXlw9v2YC4Z3GBjJAcAQAAoCia7EStWCFdeGFp2R/+MHy7p59u7vw0qUOOSI6APFVrulAY9G6bbUq3odYIAJCFW24pXT7llNLl7bYbvs9731v/eRjLD22C5AhoZ4UPi2efzTcOAED3Gzt2eNmRRyZvH01otttO+rM/a31MQMZIjoBOdcIJ1BoBAFpn5crifL01OU89Jf36143vX8DnGnKWWlfeABK0oukAHx4AgDSMGFFs0v3kk8Xy6Pytt0ozZrTunDvt1LpjAU2i5gjoFO7FHwAA0jAUGWZyhx2K8+97X3H+8MNL94l+LkXnJ06s7ZxPPFF7fEDKSI6AdsNDqQCAPP3wh605zhtv1Lf9F77QmvMCTSA5AtrBscfmHQEAAIHTT6++zd57B9Poc0YFl1xSed933inOR7vt/vGPq58XSJl5BzfRGRgY8MHBwbzDAOqT1A13XI1RB/9/AgA6XOFzacoU6fnnG9u3/HNsl12kxx6L34fPPKTIzOa5+0C17ag5AvLChwAAoJ19+9vBtN7EKGrUqNLlpMRozJjGzwG0EMkRAAAAhps1q/kv8taujS8/8URp+vTi8qpVzZ0HaBGSIyBLdLYAAOgFZ545vCw6oPmll0qPPEIvrGg7JEdAu+LDAgDQqc47b3jZe96TfRxAnUiOgKzMnFmcJ/EBAPSKDTcsXR45Mp84gBqQHAFZufrqvCMAACB7y5eXLic9hwS0AZIjAAAAtN7BBxfn118/vziAOpAcAVlIGtsIAIBudeedxfm33sovDqAOJEcAAADIxl135R0BUBHJEZClKVPyjgAAgPwcdFDeEQAVkRwBaYs2qas0yvixx6YeCgAAmXr3u/OOAKgLyRHQLq64Iu8IAABorQULivObb55bGECtSI6AVvjYx4IaonXrkrf5zGeyiwcAgHbz0kt5RwBU1Zd3AEBX+K//CqYjRyb3Rsc4RwCAXkQvregg1BwBaYo+b1QPPkgAAAAyR3IEAAAAACI5Alpvq60a39edWiMAAICckBwBrbZw4fAyEh4AAIC2R3IEpKXR540AAACQC5IjAAAAABDJEZAOao0AAAA6DskR0KxqiRDPGwEAAHQEkiOglbbcMu8IAAAA0KDUkiMzu9zMlprZ45Gy75nZU2Y238xuNrOJYflUM1tpZo+GPz9JKy4gVS+8kHcEAAAXmMFdAAAYpElEQVQAaFCaNUdXSjqkrGy2pJ3cfRdJf5B0dmTds+4+Pfw5OcW4AAAAAGCY1JIjd79P0qtlZXe7+9pwca4k2iChu/G8EQAAQMfI85mj4yXdEVne2sweMbN7zWzfvIICmkZCBAAA0JH68jipmf29pLWSrgmLFkma4u6vmNn7Jd1iZju6+/KYfU+SdJIkTZkyJauQgfqQIAEAAHSczGuOzGympI9K+rR7cAfp7qvd/ZVwfp6kZyW9N25/d7/E3QfcfaC/vz+rsIHqSIgAAAA6WqbJkZkdIuksSYe7+9uR8n4zGxnObyNpmqTnsowNAAAAQG9LrVmdmV0naT9Jk8xsoaSvK+idboyk2RYMnDk37Jnug5K+ZWZrJQ1JOtndX409MAAAAACkILXkyN2PiSm+LGHbmyTdlFYsAAAAAFBNnr3VAQAAAEDbIDkCAAAAAJEcAc3567/OOwIAAAC0CMkR0Ixrrqm+DQAAADoCyREAAAAAiOQIAAAAACSRHAEAAACApCrjHJnZY5I8ab2779LyiAAAAAAgB9UGgf1oOD0lnP40nH5a0tupRAQAAAAAOaiYHLn785JkZvu4+z6RVbPM7AFJ30ozOCBXZsV5T6xABQAAQJeo9Zmj8Wb254UFM/uApPHphAR0IJInAACAjletWV3B8ZKuMLMNFTyD9EZYBgAAAABdoWpyZGYjJL3H3Xc1swmSzN3fSD80AAAAAMhO1WZ17r5O0qnh/HISI/Sk6PNHAAAA6Eq1PnM028y+bGZbmdnGhZ9UIwMAAACADNXzzJFU7NJbCp492qa14QBt4phj8o4AAAAAGaspOXL3rdMOBGgr119ffRua2gEAAHSVWmuOZGY7SdpB0thCmbtfnUZQAAAAAJC1mpIjM/u6pP0UJEe3SzpU0v2SSI7QO8wYzwgAAKCL1dohwyckHShpsbsfJ2lXSWNSiwoAAAAAMlZrcrQy7NJ7bTjW0VLRGQN6ATVFAAAAPaPWZ44GzWyipEslzZO0QtJDqUUFtBN3Ol8AAADoAbX2VveFcPYnZnanpAnuPj+9sIAcTZ6cdwQAAADIQa0dMlwt6X8k/Y+7P5VuSEDOFi9OXhfXKcOnP51uPAAAAMhErc8cXSlpsqQLzOxZM7vJzE5PLyygg/zHf+QdAQAAAFqg1mZ1vzSzeyXtIWl/SSdL2lHSj1KMDQAAAAAyU1PNkZnNkfSApE9J+r2kPdz9fWkGBuRu882L88cck18cAAAAyEStzermS3pH0k6SdpG0k5mNSy0qoB0sWlScv/ba4jw91wEAAHSlWpvVnSlJZra+pOMkXSFpczEQLHoVCRIAAEDXqbW3ulMl7Svp/ZKel3S5gt7rgM62erW02WbS668Hy5WSHsY7AgAA6Gq1DgI7TtJ5kua5+9oU4wFab/Vqqa9PGjly+LqxY4NpXBfdcQrbkCQBAAB0nZqeOXL370kaJekzkmRm/Wa2dZqBAS0zdmyQHFVLaOpJeNyLPwAAAOgKtfZW93VJZ0k6OywaJYnBXQAAAAB0jVp7qztS0uGS3pIkd39J0gbVdjKzy81sqZk9Hinb2Mxmm9nT4XSjsNzM7Hwze8bM5pvZ7vX/OkCLHH103hEAAAAgY7UmR++4u0tySTKz8TXud6WkQ8rKZkma4+7TJM0JlyXpUEnTwp+TJF1c4zmAxkSb0ZU3j/sPKkYBAAB6Ta3J0Y1m9m+SJprZ5yT9QtK/V9vJ3e+T9GpZ8QxJV4XzV0k6IlJ+tQfmhueaXGN8QLx6nyMqiOu8AQAAAF2t1nGOvm9mH5G0XNJ2kr7m7rMbPOdm7r4oPO4iM9s0LN9C0guR7RaGZYsEtEq1XunoYAEAAKBn1dqVt8JkaLYkmdlIM/u0u1/TwljivuIfdqdqZicpaHanKVOmtPD06FkkRAAAAFCVZnVmNsHMzjazC83soLDThFMlPSfpkw2ec0mhuVw4XRqWL5S0VWS7LSW9VL6zu1/i7gPuPtDf399gCAAAAABQqtozRz9V0IzuMUknSrpb0lGSZrj7jAbPeZukmeH8TEm3Rso/GyZge0t6o9D8Dmhaee0Qg7gCAACgTLVmddu4+86SZGb/LullSVPc/c1aDm5m10naT9IkM1so6euSvqOgg4cTJP1JQbIlSbdLOkzSM5LelnRcfb8KUOaoo+LLP9lopScAAAC6mXmF5y3M7LfuvnvSct4GBgZ8cHAw7zDQDgo1QevWFefLu+pOqi3imSMAAICuZmbz3H2g2nbVao52NbPlhWNKGhcumyR39wlNxgm01ogR9SU71CIBAAAgVDE5cncGe0F3u+GGvCMAAABAm6h1EFigcxVqkmg+BwAAgApIjtB96IkOAAAADSA5QueLS4ZIkAAAAFAnkiP0LjpjAAAAQATJEXpLf39xns4YAAAAEEFyhN6ydGneEQAAAKBNkRyhu5T3SBfXQ507PdcBAABgGJIjdA8SHgAAADSB5AgAAAAARHKEbkQNEgAAABrQl3cAQFOSxjMiQQIAAECdqDkCAAAAAJEcAQAAAIAkkiMAAAAAkERyBAAAAACSSI7QLeiAAQAAAE0iOQIAAAAAkRwBAAAAgCSSIwAAAACQRHKETpY0ACwAAADQAJIjAAAAABDJEQAAAABIIjkCAAAAAEkkR+gGG2+cdwQAAADoAiRH6HyvvJJ3BAAAAOgCJEdoX2+8EfRIt2JF3pEAAACgB5AcoX1NnBhMN9gg3zgAAADQE0iO0BmOPLJ0mTGOAAAA0GIkR+gMt9ySdwQAAADociRH6BwbbRRMqTUCAABACkiO0Dlef136+MdLy9zziQUAAABdpy/rE5rZdpJuiBRtI+lrkiZK+pykZWH5V9399ozDQ7u7+ebiPIkRAAAAWijz5Mjdfy9puiSZ2UhJL0q6WdJxkn7g7t/POia0oSOOKM6705QOAAAAqcu7Wd2Bkp519+dzjgN5WblSWrVqePmtt1bej1ojAAAAtFjeydHRkq6LLJ9qZvPN7HIz2yivoJCh9daTxo2rXjMUTYZIjAAAAJCC3JIjMxst6XBJ/xkWXSxpWwVN7hZJOjdhv5PMbNDMBpctWxa3CbqVO4kRAAAAUpNnzdGhkn7r7kskyd2XuPuQu6+TdKmkPeN2cvdL3H3A3Qf6+/szDBcAAABAN8szOTpGkSZ1ZjY5su5ISY9nHhGydfjheUcAAAAA/J/Me6uTJDNbT9JHJP1NpPgcM5suySUtKFuHbvTzn1ffhmZ0AAAAyEguyZG7vy1pk7Kyz+QRCwAAAABI+fdWBxQxlhEAAAByRHKE9hMdABYAAADICMkR2k+1AWABAACAFJAcIR80oQMAAECbITlC/uiRDgAAAG2A5AgAAAAARHKELJgFP2vWDF/3sY8l77fhhunFBAAAAJQhOUJ2Ro8eXnbbbaXL0WeRXn893XgAAACACJIjpCuu4wU6YwAAAEAbIjlC7QrN45o9BgAAANCGSI4AAAAAQCRHqFW0xqfW2p/odtW666Y7bwAAAOSM5Aj5ISECAABAGyE5QmMOPbT+fUiGAAAA0MZIjtCYO++svD6pSd1HP5pOPAAAAECT+vIOAD3m5z+vbbsZM9KNAwAAAChDzRHqE60FSrNb7ltuSe/YAAAAQAySI1RXbxI0fnxxvp7njA4/vL7zAAAAAC1EcoT6HXJI5fVvv93YcW+9tbH9AAAAgBYgOUL97rijOF9eqxRdXm+9+o/tTq92AAAAyAXJEWp38MHx5UnN7t56K71YAAAAgBYjOULtot13l9fumCV33w0AAAB0AJIjVDZ2bPI6EiAAAAB0EZIjVLZ6deX1cQkSSRMAAAA6EMkRAAAAAIjkCK3gLr32WvDMEbVGAAAA6FAkR6hNtaRn4kRp3bpsYgEAAABSQHIEAAAAACI5QiVJ4xcBAAAAXYjkCAAAAABEcgQAAAAAkkiOkCTapO6gg/KLAwAAAMgIyVEnefXVbLrKLn/W6K670j8nAAAAkLO+vE5sZgskvSlpSNJadx8ws40l3SBpqqQFkj7p7q/lFWPb2WSTYJpmgnTwwaXLjFsEAACAHpF3zdH+7j7d3QfC5VmS5rj7NElzwmVI2fUcd/fdxXkSIwAAAPSQvJOjcjMkXRXOXyXpiBxj6T08ZwQAAIAelmdy5JLuNrN5ZnZSWLaZuy+SpHC6aW7R9ZryZIjnjAAAANBjcnvmSNI+7v6SmW0qabaZPVXLTmEidZIkTZkyJc342pdZ65u8zZ5dnKc5HQAAAHpQbjVH7v5SOF0q6WZJe0paYmaTJSmcLo3Z7xJ3H3D3gf7+/ixD7l5ZPc8EAAAAtLFckiMzG29mGxTmJR0k6XFJt0maGW42U9KtecTX9V55JXkdtUYAAADoUXk1q9tM0s0W1Fj0SbrW3e80s4cl3WhmJ0j6k6Sjcoqve1WqJSIxAgAAQA/LJTly9+ck7RpT/oqkA7OPqM196EN5RwAAAAB0vXbryhtx7ruv9cd0L/7stRe1RgAAAOh5JEe9JKlJ3dy52cYBAAAAtCGSo0613355RwAAAAB0FZKjTnXvvfVt3xd5vIwmdAAAAMAwJEedpJmkZmiodXEAAAAAXYjkCAAAAABEctQboh0x0KQOAAAAiEVy1O7Ke5gjuQEAAABSQXLU7ZK67wYAAABQguSol1DrBAAAACQiOepk1WqFeNYIAAAAqBnJUbeiOR0AAABQF5KjTtFMzQ+1RgAAAEBVJEfdiOZ0AAAAQN1IjrrNAQfkHQEAAADQkUiO2lnSc0OVaoPuuae27QAAAACUIDnqJnTCAAAAADSM5Chva9akc1xqjQAAAIC6kBzlbfTooMan0WSmUFtEJwwAAABAU0iO2sWICn+KaskOzekAAACAppEc5SmNpIZaIwAAAKAhfXkHgATVEieSIAAAAKClqDkCAAAAAJEcAQAAAIAkkqP80IkCAAAA0FZIjtpJXMLEs0UAAABAJkiOAAAAAEAkR/nbZ5/hZTS5AwAAADJHcpS3++/POwIAAAAAIjnKR3nNEM8VAQAAALkjOWo3++6bdwQAAABATyI5yoJZ7c8RRZvZUaMEAAAAZKYv7wC6XjQpojkdAAAA0LYyrzkys63M7B4ze9LMnjCz08Pyb5jZi2b2aPhzWNaxAQAAAOhdedQcrZX0JXf/rZltIGmemc0O1/3A3b+fQ0z5c6cLbwAAACBHmSdH7r5I0qJw/k0ze1LSFlnHkYloslNoQlco22OP7OMBAAAAkCjXDhnMbKqk3ST9Jiw61czmm9nlZrZRboGlyT34eeih6tsBAAAAyExuyZGZrS/pJklnuPtySRdL2lbSdAU1S+cm7HeSmQ2a2eCyZcsyi7ducbVGAAAAANpWLsmRmY1SkBhd4+4/kyR3X+LuQ+6+TtKlkvaM29fdL3H3AXcf6O/vzy5oAAAAAF0tj97qTNJlkp509/Mi5ZMjmx0p6fGsY2sZao0AAACAjpNHb3X7SPqMpMfM7NGw7KuSjjGz6ZJc0gJJf5NDbPlyl4aGpJEj844EAAAA6Dl59FZ3v6S4PqtvzzqWVDTbHTeJEQAAAJCLXHur6zrliRFN6gAAAICOQXLUKiRGAAAAQEcjOWoFEiMAAACg45EctRqJEQAAANCRSI5awV3ae28SIwAAAKCDkRy1yoMP5h0BAAAAgCaQHAEAAACASI4AAAAAQBLJEQAAAABIIjkCAAAAAEkkRwAAAAAgieQIAAAAACSRHAEAAACAJJIjAAAAAJBEcgQAAAAAkkiOAAAAAEASyREAAAAASCI5AgAAAABJkrl73jE0zMyWSXo+7zgiJkl6Oe8guhzXOH1c4/RxjbPBdU4f1zh9XOP0cY2zkfd1fre791fbqKOTo3ZjZoPuPpB3HN2Ma5w+rnH6uMbZ4Dqnj2ucPq5x+rjG2eiU60yzOgAAAAAQyREAAAAASCI5arVL8g6gB3CN08c1Th/XOBtc5/RxjdPHNU4f1zgbHXGdeeYIAAAAAETNEQAAAABIIjlqCTM7xMx+b2bPmNmsvOPpBma2lZndY2ZPmtkTZnZ6WP4NM3vRzB4Nfw7LO9ZOZ2YLzOyx8HoOhmUbm9lsM3s6nG6Ud5ydysy2i7xeHzWz5WZ2Bq/l5pjZ5Wa21Mwej5TFvm4tcH74Hj3fzHbPL/LOkXCNv2dmT4XX8WYzmxiWTzWzlZHX80/yi7yzJFznxPcHMzs7fC3/3swOzifqzpJwjW+IXN8FZvZoWM5ruQEV7ts67n2ZZnVNMrORkv4g6SOSFkp6WNIx7v67XAPrcGY2WdJkd/+tmW0gaZ6kIyR9UtIKd/9+rgF2ETNbIGnA3V+OlJ0j6VV3/06Y8G/k7mflFWO3CN8vXpS0l6TjxGu5YWb2QUkrJF3t7juFZbGv2/DG8jRJhym49j9y973yir1TJFzjgyT90t3Xmtl3JSm8xlMl/VdhO9Qu4Tp/QzHvD2a2g6TrJO0p6V2SfiHpve4+lGnQHSbuGpetP1fSG+7+LV7Ljalw33asOux9mZqj5u0p6Rl3f87d35F0vaQZOcfU8dx9kbv/Npx/U9KTkrbIN6qeMkPSVeH8VQre4NC8AyU96+7tNHh1R3L3+yS9Wlac9LqdoeCmyN19rqSJ4Qc5Koi7xu5+t7uvDRfnStoy88C6TMJrOckMSde7+2p3/6OkZxTch6CCStfYzEzBF6/XZRpUl6lw39Zx78skR83bQtILkeWF4ia+pcJvcXaT9Juw6NSwCvZymnu1hEu628zmmdlJYdlm7r5ICt7wJG2aW3Td5WiVfgDzWm6tpNct79PpOF7SHZHlrc3sETO718z2zSuoLhL3/sBrufX2lbTE3Z+OlPFabkLZfVvHvS+THDXPYspoq9giZra+pJskneHuyyVdLGlbSdMlLZJ0bo7hdYt93H13SYdKOiVsfoAWM7PRkg6X9J9hEa/l7PA+3WJm9veS1kq6JixaJGmKu+8m6W8lXWtmE/KKrwskvT/wWm69Y1T6pRWv5SbE3LclbhpT1havZZKj5i2UtFVkeUtJL+UUS1cxs1EK/sGucfefSZK7L3H3IXdfJ+lS0Zygae7+UjhdKulmBdd0SaF6O5wuzS/CrnGopN+6+xKJ13JKkl63vE+3kJnNlPRRSZ/28MHlsJnXK+H8PEnPSnpvflF2tgrvD7yWW8jM+iR9XNINhTJey42Lu29TB74vkxw172FJ08xs6/Cb4aMl3ZZzTB0vbAN8maQn3f28SHm0PeqRkh4v3xe1M7Px4YOTMrPxkg5ScE1vkzQz3GympFvzibCrlHw7yWs5FUmv29skfTbsHWlvBQ9eL8ojwE5nZodIOkvS4e7+dqS8P+xwRGa2jaRpkp7LJ8rOV+H94TZJR5vZGDPbWsF1fijr+LrIhyU95e4LCwW8lhuTdN+mDnxf7ss7gE4X9thzqqS7JI2UdLm7P5FzWN1gH0mfkfRYoXtNSV+VdIyZTVdQ9bpA0t/kE17X2EzSzcF7mvokXevud5rZw5JuNLMTJP1J0lE5xtjxzGw9BT1aRl+v5/BabpyZXSdpP0mTzGyhpK9L+o7iX7e3K+gR6RlJbyvoKRBVJFzjsyWNkTQ7fN+Y6+4nS/qgpG+Z2VpJQ5JOdvdaOxnoaQnXeb+49wd3f8LMbpT0OwXNGk+hp7rq4q6xu1+m4c+BSryWG5V039Zx78t05Q0AAAAAolkdAAAAAEgiOQIAAAAASSRHAAAAACCJ5AgAAAAAJJEcAQAAAIAkkiMAQBsxsyEzezTyM6uFx55qZownBQBIxDhHAIB2stLdp+cdBACgN1FzBABoe2a2wMy+a2YPhT/vCcvfbWZzzGx+OJ0Slm9mZjeb2f+GPx8IDzXSzC41syfM7G4zGxdu/0Uz+114nOtz+jUBADkjOQIAtJNxZc3qPhVZt9zd95R0oaQfhmUXSrra3XeRdI2k88Py8yXd6+67Stpd0hNh+TRJP3b3HSW9Lukvw/JZknYLj3NyWr8cAKC9mbvnHQMAAJIkM1vh7uvHlC+QdIC7P2dmoyQtdvdNzOxlSZPdfU1YvsjdJ5nZMklbuvvqyDGmSprt7tPC5bMkjXL3fzazOyWtkHSLpFvcfUXKvyoAoA1RcwQA6BSeMJ+0TZzVkfkhFZ+9/QtJP5b0fknzzIxncgGgB5EcAQA6xaci0wfD+V9LOjqc/7Sk+8P5OZI+L0lmNtLMJiQd1MxGSNrK3e+R9BVJEyUNq70CAHQ/vhkDALSTcWb2aGT5TncvdOc9xsx+o+CLvWPCsi9KutzM/k7SMknHheWnS7rEzE5QUEP0eUmLEs45UtJ/mNmGkkzSD9z99Zb9RgCAjsEzRwCAthc+czTg7i/nHQsAoHvRrA4AAAAARM0RAAAAAEii5ggAAAAAJJEcAQAAAIAkkiMAAAAAkERyBAAAAACSSI4AAAAAQBLJEQAAAABIkv4/EEoD1RvaskwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1008x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "env = gym.make('CartPole-v0')\n",
    "agent = Agent(4, 'cartpole')\n",
    "agent.train()\n",
    "optimiser = optim.Adam(agent.parameters(), lr=1e-3)\n",
    "discount = 0.99\n",
    "trace_decay = 0.95\n",
    "value_loss_weight = 0.5\n",
    "entropy_loss_weight = 0.01\n",
    "epochs = 200\n",
    "rollouts = 32\n",
    "total_rewards = []\n",
    "\n",
    "plt.figure(figsize=(14, 6))\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Reward')\n",
    "\n",
    "# Converts numpy observations from the environment into a batch Tensor for PyTorch\n",
    "def obs_to_tensor(obs):\n",
    "    return torch.tensor(obs, dtype=torch.float32).unsqueeze(0)\n",
    "\n",
    "def plot():\n",
    "    plt.plot(range(len(total_rewards)), total_rewards, 'r-')\n",
    "    clear_output(wait=True)\n",
    "    display(plt.gcf())\n",
    "\n",
    "for _ in range(epochs):\n",
    "    optimiser.zero_grad()    \n",
    "    total_reward = 0\n",
    "    for _ in range(rollouts):\n",
    "        obs, done = obs_to_tensor(env.reset()), False\n",
    "        values, log_probs_action, rewards, entropies = [], [], [], []  # Collect outputs for training\n",
    "        while not done:\n",
    "            # Sample an action from the policy conditioned on the current observation\n",
    "            policy, value = agent(obs)\n",
    "            action = policy.sample()\n",
    "            # Take a step in the enviroment based on the action\n",
    "            obs, reward, done, _ = env.step(action.item())\n",
    "            obs = obs_to_tensor(obs)\n",
    "            total_reward += reward\n",
    "\n",
    "            # Store outputs that will be used for training\n",
    "            rewards.append(reward)\n",
    "            values.append(value)\n",
    "            log_probs_action.append(policy.log_prob(action))\n",
    "            entropies.append(policy.entropy())\n",
    "\n",
    "        # Initialise the final reward and generalised advantage at 0\n",
    "        ep_return, gae = torch.zeros(1), torch.zeros(1)\n",
    "        values.append(ep_return)\n",
    "        trajectory_length = len(rewards)\n",
    "        loss = 0\n",
    "        for i in reversed(range(trajectory_length)):  # Calculate the return backwards\n",
    "            ep_return = rewards[i] + discount * ep_return  # Calculate the return recursively backwards\n",
    "            advantage = ep_return - values[i]  # Calculate the advantage (used here only for the critic loss)\n",
    "            loss += value_loss_weight * advantage ** 2\n",
    "            td_error = rewards[i] + discount * values[i + 1].item() - values[i].item()  # Calculate the TD error\n",
    "            gae = gae * discount * trace_decay + td_error  # Accumulate the TD error into the generalised advantage\n",
    "            loss -= log_probs_action[i] * gae  # Use the generalised advantage to weight the policy gradient\n",
    "            loss -= entropy_loss_weight * entropies[i]  # Encourage higher entropy policies\n",
    "        loss.backward()  # Accumulate gradients\n",
    "    optimiser.step()  # Perform batch update\n",
    "\n",
    "    total_rewards.append(total_reward / rollouts)\n",
    "    plot()\n",
    "\n",
    "clear_output(wait=True)\n",
    "display('Average final reward: %.2f' % total_rewards[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAABA5JREFUeJzt3NFJA0EUQNGspArbSB22EWsybViHbdjG+BNUiB/ibpjleg4EQiDD+7q8hEmWMcYBgJ6H2QMAcB8CDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRx9kDXPk5LcCtZc2bbfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFHH2QMAcDi8XZ5vXjudX1adaYMHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gB1Y+8+RPxF4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAO1qW5dePrQk8QNRx9gAAfHl9P38+P608ywYPsBPf474FgQeIEniAKIEH2Imnx8um5y1jjE0P/KNdDAGwtTXXH8cYq+5O7uIWzT3ufwL8d7sI/E4+RQBsbuYC6zt4gCiBB4gSeIAogQeIEniAKIEHiBJ4gKhd3IMHqJr5Ox8bPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxAlMADRAk8QJTAA0QJPECUwANECTxA1HH2AFfL7AEAamzwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUR+epByFZ7bXeAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.axis('off')\n",
    "\n",
    "agent.eval()\n",
    "obs, done = obs_to_tensor(env.reset()), False\n",
    "view = plt.imshow(env.render(mode='rgb_array'))\n",
    "while not done:\n",
    "    view.set_data(env.render(mode='rgb_array'))\n",
    "    display(plt.gcf())\n",
    "    clear_output(wait=True)\n",
    "    with torch.no_grad():\n",
    "        obs, _, done, _ = env.step(agent(obs)[0].sample().item())\n",
    "    obs = obs_to_tensor(obs)\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Average final reward: -1121.73'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1UAAAF3CAYAAABNBGHSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm8HFWZ//HvE8K+yBrIECKoQQQElCuLLCoCAiphFRhF3IggOKP8VDZ1XEdQGB0U0SgqCIggsgkYAiKbbIksITBIQNRAhLDJKpDk+f1R1Xbdvr1XVZ+q6s/79epXVZ/anq7uW7eeOqdOmbsLAAAAANCfcaEDAAAAAIAyI6kCAAAAgBRIqgAAAAAgBZIqAAAAAEiBpAoAAAAAUiCpAgAAAIAUSKoAAAAAIAWSKgAAAABIgaQKAAAAAFIgqQIAAACAFMaHDiCUNddc09dff/3QYQAAAAAoqNmzZz/u7mt1mm9ok6r1119fs2bNCh0GAAAAgIIys790Mx/N/wAAAAAgBZIqAAAAAEiBpAoAAAAAUiCpAgAAAIAUSKoAAAAAIAWSKgAAAABIgaQKAAAAAFIgqQIAAACAFEiqAAAAACAFkioAAAAASIGkCgAAAABSIKkCAADlZxa9ACAAkioAAAAASIGkCgAAlJt76AgADDmSKgAAUG6rrx46AgBDjqQKAACU29NPh44AwJAjqQIAANVx1VWhIwAwhEiqAABAdeyyS+gIAAyhIEmVme1vZnPNbImZjTRMO9bM5pnZfWb2rkT5bnHZPDM7JlG+gZndYmb3m9kvzWyZQX4WAAAAAMMtVE3V3ZL2kXRdstDMNpZ0oKRNJO0m6ftmtpSZLSXpVEm7S9pY0kHxvJJ0oqRvu/sUSU9J+uhgPgJKo/bsEp5fAgAAgBwESarc/V53v6/JpKmSznX3l9z9z5LmSdoqfs1z9wfd/WVJ50qaamYmaSdJv4qXP0PSXvl/AgAAAACIFO2eqnUl/S3xfn5c1qp8DUlPu/uihnIAAAAAGIjxea3YzK6StE6TSce7+8WtFmtS5mqe/Hmb+VvFNE3SNEmaPHlyq9kAAAAAoGu5JVXuvnMfi82XtF7i/SRJj8Tjzcofl7SqmY2Pa6uS8zeLabqk6ZI0MjLC49cBAAAApFa05n+XSDrQzJY1sw0kTZF0q6TbJE2Je/pbRlFnFpe4u0u6RtJ+8fKHSGpVCwYAAAAAmQvVpfreZjZf0raSLjOzGZLk7nMlnSfpHkm/lXSEuy+Oa6GOlDRD0r2SzovnlaSjJR1lZvMU3WN1+mA/DQAAAIBhZlFlz/AZGRnxWbNmhQ4Dg5DsSn1If+8AUGmNj8zgWA8gI2Y2291HOs1XtOZ/AAAA6Xz+86EjADBkSKoAAEC1fP3roSMAMGRIqgAAAAAgBZIqDJfGdvcAgHLj/ikABUBSBQAAymv11UNHAAAkVQAAoMSefjp0BABAUgUAACpi5szQEaAbTzwhvfJK6CiATI0PHQAAAEAmdt45dAToxpprRkPuh0OFUFMFAMAwMYteNJsDgMyQVAEAMIxWWy10BBhGBx0UOgIgFyRVAAAAGIxzzw0dAZALkipUG8+lAoC6YTomvv/9oSNAJ6edFjoCIDMkVQAAoPwam5Wdc06YONC9I48MHQGQGZIqAABQfiRR5bNkSegIgMyQVAEAMAw22GBs2e67Dz4OAKggkioAAIbBQw+NLfvtbwceBgBUEUkVAAAAAKRAUoXhM0y9XwFAI/fQEeSrCJ/PPfpfc+ONoSMBMCAkVQAAVF3jxaQiJB5VNi4+vdp++2jf/+EPYeMBkDuSKgAAgKzst9/Ysu22o5UEUHEkVQAADIvzzx9btuKKg48jKzNnho5grAsuqI+/9rWjpy1aNNhYiqbxWWJAhZBUAQBQZUccUR9vVovywguDiyVre+0VOoLRXvWq+vgNN0jz5o1uarn00oOPqUjOPTd0BEBuSKoAAKiy738/dAT56ZQQvu51g4mj5pln6uPbbTfYbZfVaaeFjgDIBEkVhgM3ZQO9e/Wro/tAuBekmtZcM3QE+XvggcFtK/l3EuJ/Tln/Vo88MnQEQCZIqgAAzf31r6EjGA6hToYXLhz8NqvqhhvaTx9kklW2xGrJktARAJkgqcLwoLYKQNG84Q2D29YwHQOTnzXvJGPHHaUddqi/J0nobNNNQ0cAZI6kCgCAUP7v/0JHEClb7UZRmEnXX19//41vdN6Xq62Wb0ySNG1a/ttIY86c7ufdZJPhuiCA0hofOgAAAJCTd70rdAThuNcTHLPsT8z7faDy009nG0czP/qRNH16/ttJY8IE6bHH2s9T28fjxpFYofCoqQIAoKquvDJ0BMMhzQn/e96TPukregLVTLKGr5m11x5MHEBGSKoAABhWu+0WOoJsHHdc8/K87q1KJgTdJkN/+EPz8ssui4bjUpySHXbY2LKiN+nccMP20zvVYgEFQ1IFDPsT7gGkV+vBr+gnso2uuKI+bib9+tfhYknj618f7PZ23LH3ZbbddmzZVluNft/v76dqTePK9ncEiKQKw2r11aOhWfSEew7g5VM7gX396zvPW7UTjhD4G8lfnvu43d/Am99cH9933/ImVq0kP3un2pFBOfDAaHjbbWOnffaz6dad/LxFehbZ4Yd3N1/y7+CrX62Pd3OsBwIKklSZ2f5mNtfMlpjZSKJ8DTO7xsyeM7PvNSyzpZnNMbN5ZnaKWfRXZ2arm9lMM7s/Hg6gWx2U3lNPjS3jpDG8fq72/+lPnecZN47vF2hl9mzpk5+sv993X+mCC8LFk6f770+/jn6a/jX65S9Hvz/ooPr4SSf1t85mnngiu3Wl9YMf9L7M5z9fH+/mWA8EFKqm6m5J+0i6rqH8n5K+IOkzTZY5TdI0SVPiV60h+DGSrnb3KZKujt8D0t57h44AeTnqqP6WI7HCMNlii+7nPeWU0YnUfvtlH09Iydq4tPpp+tdM8nh0zjnSzTePnrbHHv2v+53v7H/ZEE47LRr+8If1sk4J64MPlrPJLSorSFLl7ve6+31Nyp939xsUJVf/YmYTJa3i7je5u0s6U9Je8eSpks6Ix89IlGPYXXRR++n77ju2jINzOXz7293Py3fan299K3QE5dHvbyxNxwTduPPO3ubfZ5/RiVWV/nZmz66Ph/xcn/pU62lbby2ts079/RVXRLFusknv27nqqt6XCenII6Nhsw43ZsxovsxrX5tfPEAfynJP1bqS5ifez4/LJGltd18gSfFwwoBjQ1lV7b6Bstt889ARIOlznwsdQXZOPjk6OT3hhMFsr9uT9iLe67fPPqPfX3hhmDiK6r//uz7ez/fX7IJQcj0LFkjXXjt6+j331Gtkjj22/fqnTu09pnaefDLabrMm81lasqT1tF13zXfbQEZyS6rM7Cozu7vJq5+/+Gb/oXo+mpnZNDObZWazFi5c2EcYqKy8ut1F9+66azDbGZbvl2YxdZ+JW5R3OiEtukH1MJg8HjYmWWV29dXp13H88enX0cmOO0bfwS23jJ12wgntv/9OLTR6tcYa0bDWuVMWNt208zy9Jqwnn9xfLECGckuq3H1nd9+0yeviPlY3X9KkxPtJkh6Jxx+NmwfWmgm2fLCBu0939xF3H1lrrbX6CAOVVMSrxWiNRKEz9tHw2WWX1glXP8e4ZE1+UX9PM2f2Nv9OO9XHi/qZkrbaKvru3KXllut/Pbffnl1MWZgzp3n55Zd3XnaFFaLhgw+OLv9Ms1vxEdQRR0gTJkjPPRc6koEpRfO/uFnfs2a2Tdzr3wcl1ZKzSyQdEo8fkigHekNtVbG0u/egH8OSPPPbHayf/aw+nubEt5Nf/GL0+8bvOXkPzSc+kX57jR39FPF3tdeAb6FO1tZkdTzpdj0vvtj/NrPspCNrExJ3bLz73Z3nf/HFaMj9VMX3/e9LCxeOPXZVmHmAEw0z21vSdyWtJelpSXe4+7viaQ9JWkXSMvG0Xd39nrjr9Z9JWl7SFZI+6e5uZmtIOk/SZEl/lbS/uz/ZKYaRkRGfNWtW1h8NRZI8CWiXMLWaNiwn4UXQ6oSt2XdQm/dNbxp9BbbV91Wb371a3++iRdEz1qT6Z2m2H1/zGumBB3pff1Y1HqHl/Tkaf1PJ31u3y3azXLvP0Wza6qtH98N0G0unGJttN7R+/557+Y6y2F6jLbeU/vjHdDE0LtesPIt4k+uYM6e7pnud1pWM5U9/av78qcZ4x42rlzUey1stg3DOPlv6wAei8fXXl/7856DhpGVms919pON8IZKqIiCpGgIkVeXRbVLV6gS22bz9zF8mvdQc9PNZq3LSUuWkSpLOPbf+INlW0n7eov4W0iZVvSy30krS88/3vq2s1WJ/3etGP3NrEElVmvUk19XuuN5qGw88EH3m2vTaMg88UK+1KsJvEpFVVpGefTYaHzdOWrw4bDwpdZtUlaL5H5AbDsLFkvf3kVx/lXq3a5Tlftxww/p4EZuA5a3bTo222270+8Z9ZSZ98IP9xdDu5DiZUOX191O7r6eKuv1N1xKqopg3L3QEg5Vs7pe8n+o1r6mPl7w2pDJeeaWeUC27bNSz40MPBQ1pUEiqAIQV6kS9Ks9hSp7wTpgwuolMFu4b80jB4liyRPrnPzvPJ/VfWzdhQne/0RtuaL8eSfr5zzuv5+yzO8/T7LPUnu+TZ/JTlftOO9WUNCp6Dff06c3LTz99sHG0c/jh2a2r1f1UtZoshFXrMXSNNaRttonGP/axcPEMEEkVkFTEf5gYq9fvadVVm5eX9cSwVXOsRx9tPv83vpFvPJ0cf3wUc7cJUCvJLsXNpKWWkpZfvv38vZT3E0/Wy9buQ2gm2Xtdo9NOq48n/z5qV4wxWmOPemblbaJ0xBHNyz/ykXTrbfdb7NUPfpDdulpp96wrDE6tF8fzz5d+/ONo/MYbR89z663RvYXPPDN2+RdekL77Xemxlp15FxZJFYYXCVSx/Md/RMN+btTvJPngymZXqcuaXHXruOPCbr/2wNTll5f+93/7W8egv6Nmzfe6sf32o9/XuoCu6aa2qtV2k89ZSv6Om51MLlkSvVZaqbftdVKV2iop6knuQx+qvx8/fuw8Ra+lkqJOa/LQrNZ0zz3z2VZNL/u41gHP+uvnEgr6cMcd0XHHTHrHO6Law9oFteTvdOedo85aTjpp7DpmzIjOB+6+e3BxZ4SkCtXX2O1ule8PKLN+T7b70ez7L8IV/UE93DWkT31KeuMb062j8W+40/6q3Xd00EHdrT/N/r/++tHva11A1/R7X1Uztf3QLN6q/46y8tOfSsnnViaT4GuvHXw8nYS+QHLppeG2vfLKo9/X7qfiXqriqHWLn7y4tN560fArX4mGxx9f/3972WVj1/HrX0e9l+64Y35x5oSkCtV34YWhI0AInU4oG0/MV1kl33jaecc7+jsB7nRxoFZDlFbW95/dfXdvn7ef2oIf/rA+XntOyjnn1Mta3YfSj0FcpCnShaAi1FYlt5v2mXbJZkbJJPjtb6+PF2X/f/3r/S335S9nG0cIzZqKoVgeeSQaJmvVjz8+Gv7gB9LLL0snnBC9f9WrpLvuGl3T/vLLUeI+dWrzmuOCI6kCWuEqb/762cdbbJHduorATPr973ubv1vHHttzOE195jPZrKfxxLSbE9V/+7fW09Zdt/W0WscNrXz8483LGxO4Xrs4bzXPmWf2vp6QiX6RNdbWfPvb6dfZmCgmv5cqJCRf+lK65efMySQMHXro2LLayXMWTVU32CD9OtCf2sW38ePrz1CUpGnTouHChdI73xklUTvtFLUiWrQouveq5pprpH/8o97ZRcmQVAEormYnnMkH/pbdpEmj3yevjGedJPZbM5TWd74z+n2nWo7nnhvdDHLBgubLStL8+fXxZEcN/UrGM3ly++m9Ovjg5uW1q7QTJ46dVoQmqa0kv4u0HZD0KtnxSt41SBttJH3xi/luo+juuivdQ3/XWKM+3qyG+JVXou+xl9978hldSUPSdXchnXxyNNxqq7HTagnzDTdEz6268krp6KOjslNPrc/3619H8+68c76x5oSkCkC1jXR8Xl84Dz9cH3ePrtJ166tfzT6ePHz6073N33jfRM33vtd+uU98onl5Y69jyWaBySSpMWH6y19ab+unP62P/+Qn7eOSuuvCu9Zspoxa9a6Zh+S+mzEj23U3S9DuvTfbbeRp6tR81tt4D2SyFqIbTz6ZXSzPPhvVbjR2n77UUtltA/2pNaP92c/GTttjj/r4l74UfV9veEPUBPDWW6PyxYuliy6K7stK9sxZIiRVQLcau3PmRvDBSbOfb7ut/fRQ90okP9N//Vfvy3z+890ts+KK3cfUjWa9NfVrs826n7dVt9HNJPdTYzO/WlOU5LyNv6/G30Tj836SXVV/+MNjt5/sUS4L7ZpAFsFLLw1mOzvsMPr9rrtmv43khY2i3EfVSuMFg4suGjvPMstkv91+exvMIkFdaaXmCVSrmisMTu3vZcqUsdNqF7OWWUb6whfq5TvtFB0/rrpK+sMfosSspE3/JMm86AeNnIyMjPisWbNCh4E81U6Uev2Nt7opvt2J/ZD+HaXWzb5u7N2s3VX/Zuvo5rvp97fSry99afR9Gu0+e1K/3Tv3+vmabSfttrtdVz/fReMynWL99relo45qvq5O91Btu610003dz5+c56MfHVuz1e43O+jfZa8G3d14Gbo3H4RWx8dOv8V+/xfWlnvVq+qdRaQ5luSl6H8vVXbXXdLmm0dN+1o97+33v5fe8pbRF/puvDHqKXCPPaQNN4yacS9c2LrFQiBmNtvdOzZ7oaYKaNTpgLz//hy0s5ZmfzZbtug1iK0Sqmbvs1T0/fLd76ZbvrHWqdW+/PSnm3dC0c2+7yaharXOxhovoKiadUzxj38MPo5+9FKrjWx87GPRcJ11Ws/z9rePbTmx3XbRYwyuuy66n2rXXQuXUPWCpAro1XnnhY6g+lqdrLbq+S8rg0g6ktt429v6W0cZk/puYq49AHqQaslV1vs07TqLngCj2rK+F3WQx6zvf39w20LkzjujYT89xW67bdRB0V//WuqmfxJJFdCdZic4ZTyxLbtuev4r0/fSqSv1nXaqjxf9JLtWS/TCC6EjieSRJOW5/qSi30NVU6a/NfTm5ZfTr2PQx6zVVx/s9lBX+7302jGRVL+YNm6c9N73ZhdTACRVQBaKfsJbRCuskP06zaIDc03ZT/p66Q0wD/3sv246xmjsTSytMn3PO+7YeZ5kN/JAo9VW62+5UM9wGlTPfE88MZjtYLS0Cfh73hN1YLHaaqO73y8hkipUU7//dBqRLOXnxRe7m6+q38EgE4HXvjb9OvqJt9V3d9dd6WLpJO99++Y397/stdfWxzfaqPP8Z5/d/7ZQTckuymfP7n65LJ/h1KmHy2RPf/32FpjG1lsPfpvD6rOfjYb9XigdNy56APDMmdnFFAhJFarp6acHs50DDhjMdtC7Xk6s8zgJb9YMLosEsZ9Y581Lv912sr662EtX642+9rXs4mhUu0+qlxPZdmr3IbTz7/+ezbZQTdtsk+/63/3u5uVnntl+uY03zj6WXtSefYT8/fzn0bDfe4Qlac89pTe9KZt4AqJLdVRT2m5ce+mylm5++9Npv/VyH1vjvFtuKfX6951ld7zddhXfbTxZ/May6GK+U3lNv12nD3uXyGW8dzNEl9nLLdd9TXdV9Xt87LUr9H4f6xDq/+LWW9cTqqL/7VRF7bt+4onK3tdGl+pAVt73vtARQOqt5780F0zyvvIrSV/8Yv7byFOnm4mr2mQTxdDsIbdoLeue/Hox6E5XbrllsNtDXUUTql6QVAGdnH9+6Aiqrdurie16/kuu4//9v3TxpP2n3JhQ1N4ny5PPqRq0LBKe3/ymt/m7vaI97M46K3QE5fCud4WOoFxuuy3cth9+ONy2S97pQSn87nfRcBzphERSBTQX+n4c9O+kk8Jt+wtfyH6dg25G0+oeirylffBvFbz//aEjAKoj2aEH8nHkkdFw/fWDhlEUJFVAlrjqnp1eE4g8HuDaq2QnCa3upSq6yy9vPa3Z/k3ekJ5m/4d48G/Rhf49d6NMv210Z4cd0i2/0krZxNGvT34y7PaHyZ/+FA2/8Y2wcRQESRXQrTKc4KB/Rx+dbvnkyWW7E82q/I5q7eeTXScncbI9HJYsCR0BpO57Xuvm8Qo33NB6WjfPnHr++e5iycspp4Td/rB46SVp8eJofL/9wsZSECRVQBbKWisRShH30QknZLeu2olmmgTqHe/IJpaarK8eP/VUtusD0L/f/767+R58MN12PvWpdMsP2vjxoSOorg03jIbLLss9VTH2AoDiC92cpJPPf775eBq1G4BrvvKVdOt79tl0y3fyhjf0Nv///m8+cQAYq11HP2bRa999O6+nl3tWW9ViD1KtJgXZ++tfo+H114eNo0BIqlAetQP/LruEjqS5Aw+sj0+cGC6OKnrmmcFur9eatK9/vT7+1a+Onpasrfrc5/qPKY9OMPrRqvbtnnvaT2/U6h6qNA/+BYbFaaf1Nn/ykRStjm+//rV0113199009Wtno43SLZ/GNdeE2/Yw2HnnaDhunPSWt4SNpUBIqlA+V10VOoLmfvGL+vjf/x4ujiqpdT5RxOaCvdhyy+ihlCeeGDqSbHXzvfTaBPLOO/uLpWo+8IHQEfSu7H+nZXLYYb0v06wpXON3tvnm9fF2tVutFOXemre/vT5+9tnBwqisq6+OhrQ4GIXGpqi2rDoF2H//bNaD0ZZfPnQE6W2/fX281e8tzcOI82DW+W/jm98cTCwYrSodmWSNGsz0XnmlnkQtt5z0sY/Vp2222ehaKkl64xt738YFF/QfX14+8AEeV5Cl446rj9e6VIckaqpQFbWmgXldKT3vvO7mo8OK3rzwQugIWuv2+7vxxvxiqN0HFaKHtc9+dnDborcudDJnTugIquWll6RTT62/v/PO6nVFzrOT8lHrPj15ywMkkVShirJKZrhiPHwav/PQifFKKxWz+WPjfqp1r95qeif/+Z/p4gHQnWZ/m3fcEQ1POUVae+1stvOd72SznjT+/OfQEVTPOefUx5O3PEASSRXQ3pIl/T2Etoaeh8qnWWLV+KpZZ53Wyw2TJ54YW1aEhzFjMF71qtARII3kfVR//3v0uIRXXkm3zqJdKCnahamyOuqoaLjqqmHjKKggSZWZ7W9mc81siZmNJMp3MbPZZjYnHu6UmLZlXD7PzE4xi/5CzGx1M5tpZvfHw9VCfCZUVNoDMc/IGGuFFUJH0FmnZKD2u3j00fxjAYru6adDR4BerLVWfbzZsW7VVfv731WELtSRr9rf+iabhI2joELVVN0taR9J1zWUPy7pve7+RkmHSPp5YtppkqZJmhK/dovLj5F0tbtPkXR1/B7DqvEfBFfKi+fFF0NH0J1uE6sy4e8BwGOPSRMmSD/5Sbr1bL316Pcbb5xufXngmJetl1+Ohu97X9g4CipIUuXu97r7fU3Kb3f3R+K3cyUtZ2bLmtlESau4+03u7pLOlLRXPN9USWfE42ckylFle+5ZH29sesJBFFmpNWFLvlrNVwW9JIru0T/Yqnx2YJg8+qj04Q+nW8ett2YTy6BMmhQ6gvKrHe+nTQsbR0EV+Z6qfSXd7u4vSVpX0vzEtPlxmSSt7e4LJCkeThholAjj0kvr480eDFu7F6qMtQlV9clPVuP7IImoW3rpdMs37stll023PmCYdLrYE8oGG4SOoLmHHw4dQXUst1zoCAoptxs+zOwqSes0mXS8u1/cYdlNJJ0oaddaUZPZej6KmNk0RU0INXny5F4XR0i9noyHPnknoRut2b4o2olAL5Lf7+c+FzaWfq28cr3L9qL45z9DR4C0unkGGqrtwQdDRzDaWWeV82HaKJ3caqrcfWd337TJq1NCNUnShZI+6O4PxMXzJSXrbSdJqjUTfDRuHqh4+FibmKa7+4i7j6yVvFETwOBsumk1TrpqV4hPPDF0JP157rnQEQC9O4bbptGj5IN/v/zlcHGg8grV/M/MVpV0maRj3f1fT9SMm/U9a2bbxL3+fVBSLTm7RFGnFoqHbZM2YOCGvcYq+fndeYgngP7VHjwK9ONLXwodASosVJfqe5vZfEnbSrrMzGbEk46U9DpJXzCzO+JX7R6pwyX9WNI8SQ9IuiIuP0HSLmZ2v6Rd4vcYRlWo/QCKoKy1bwCAfNQeEj3sF4rbCPIQHXe/UFETv8byr0n6WotlZknatEn5E5LemXWMADC0ynqfGMKZOFFasCB0FBiElVYa23y46MeMceOiDqzQv5NPjobjCtXIrVDYMwAG44gjQkcAiRpd5OORRzrP0w2z+gvFdPrpY8u+9a3Bx9GLAw4IHUH53XxzNFxxxbBxFBhJFcqtyP94OXkd7XvfCx0BACCtMj749ZxzQkdQfrWa6HXXbT/fECOpQnkVOaFChO+oPL75zTDbrT1TjosQ1bHNNqEjAJC1F1+MhttvHzaOAiOpAoBhlUx6jz46fAyohltuCR0BgKzV7kk77riwcRQYSRXKZaWVQkfQH04cAQBVt9NOoSNA3tZfP3QEhUVShXJ59tnm5TQdKja+HwD9OvbY0BGgW1dfHToCIBiSKpTPe98bOgIAQKO8Lp6cwOMnkaFDDw0dASqKpArlc8kloSNAN2jyWFzUHKIM3vjG0BGgin7yk9ARoKJIqoA8cfKKsuE3iyxcemnvyzR2lnL33dnEgvzMnRs6gt7xEODePf106AhKgaQKAABka889e18mVLf+6N/uu0v33BM6CuTtlFOiIS1Q2iKpAgZlWA9G1HwU27D+LgGk97e/SSMjoaPoDse6/tVqnpddNmwcBUdSBQAAgP7UHgpbdAceGDqC8nrwwWi4+uph4yg4kiqUXz/NTNC7xYujK33dXO3jimC5vOpVoSNAVdCREIrqnHNCR1BetcfZbL552DgKjqQKxdfpBP3iiwcTx7AbP74+XkuuzGjeVwXPPBM6AlRFFo+8mDw5/ToAZGfRomj4kY+EjaPgSKqAvE2cGDqCfI3jMFJKJMMoqr/8pXn5DjsMNg70ZuWVQ0eAvNT+X+y3X9g4Co7xD3EDAAAgAElEQVSzIZRTmU4IH3kkdATZ63b/l+l7ApCtrJsBX3ddtutDtm6+OXQEQFAkVSgvd07aQzr88Obl3E8FANW23npjyzbeePBxAAVCUgWgP9//fn38E58IFweys/vuoSMARuMiTTFdcUXoCNI59NDQEaCCSKqAInn5Zemll0JH0V6z2sHTTht8HMje5ZeHjgBVcNRRvS+z6abZx4H8bLJJ6AjS+elPR7+n1QsyQFIFDFKnq67LListt1w1rs7yTwoYTief3Psyc+dmHwfQyuLF9fFJk6IOl6rwfzcPtR6W2T8dkVQB6F/jfVUcdAEAZfLww6EjKLYf/SgaJh+rgqZIqgD0L3lfFTcplw+1icgbF1qAcrvzzmhIl/kdkVShPFZaKXQEw6ubK1T33lsfJ8ECkAYJP/IwdWroCMrniSei4frrBw2jDEiqUB7PPhs6gsEq0hXeZPvzbnB/BDDcLr20v+U+97ls4wCSLrpo9Ps//GH0+9VWG1wsZVHrPOvd7w4bRwmYD+nVoJGREZ81a1boMNCNWnJR5t9qMkFq9zkaE6mifOZO8Rc1bnTW7W8T6FUvv61mx3mOK8VW1u8n+VtrdvGyLJ9jUGr76KmnpFVXDRtLIGY2291HOs1HTRUwCBykAQBV9T//EzoC5G1IE6pekFQByBYPAi6X7bePhs88EzYOVM9GG4WOAIPy6U+HjiCdc88NHQEqgKQKQLZOPTV0BOjF9ddHNan07ISsJTuv2WuvcHEAnRxwQOgIUAEkVUCRFamzCql1M0aaNwJop/YA0WaKdpzDcNh66+bl9DRcd+GFoSMoFZIqYNBanUAcfPBg48iaO8kVAKAcbr21efnzzw82jiL75Cej4bLLho2jJEiqgKI466zQEQBAtpIXWn7zm3BxAK0stVQ0HOnYudvweeSRaLjTTmHjKAmSKhTbsDYLocYHQNW897318UWLpCVL2s9f60QFyNOf/xwNb7stbBxFVDsXueCCsHGURJCkysz2N7O5ZrbEzEYS5VuZ2R3x604z2zsxbTczu8/M5pnZMYnyDczsFjO738x+aWbLDPrzAJV22GGhIwBQFWbRa+mloxqCyy9vPe/119fHJ0zIPzb0Zvr00BFkY731QkdQfMsvHzqCUghVU3W3pH0kXdekfMTdt5C0m6Qfmtl4M1tK0qmSdpe0saSDzGzjeJkTJX3b3adIekrSRwfxAYCBCV1b98Mfht0+gHJrV/P+7nd3N9+jj2YXD7Jx6KHlvJd25507z7PCCvnHUXQPPxw6gtIJklS5+73ufl+T8hfcfVH8djlJtb/UrSTNc/cH3f1lSedKmmpmJmknSb+K5ztDEv22AgBQJLWT79pr7707LwPkYebMzvO8+GL+cRRd7W+0ds8ZOircPVVmtrWZzZU0R9JhcZK1rqS/JWabH5etIenpRCJWKweKp2xX8wAgL7/+9dhjIsdIDNo664x+/9a3holj0F58UVq8uP08t98eDadMyT+eisgtqTKzq8zs7iavqe2Wc/db3H0TSW+RdKyZLSepWfsnb1PeKqZpZjbLzGYtXLiwl48DDF7yBCN0E0AAyIO7tOGG0nLLNT/OlbF5GcpjwYLR72+8MUwcg7bCCtL48dJdd7WeZ1FcX3HmmYOJqQLGt5toZnPUJklx983aTOui0Wpr7n6vmT0vaVNFNVDJOwknSXpE0uOSVjWz8XFtVa281TqnS5ouSSMjIxylUS5mnFwAqJ77xtwNAIT38MPSuhVv/LT//p3//t7ylsHEUgFtkypJ74mHR8TDn8fD90t6IetgzGwDSX9z90Vm9mpJr5f0kKSnJU2Jpz8s6UBJ/+7ubmbXSNpP0X1Wh0hq89h2oASSiZN7cWqpSOgAAFXQzf+zyZM7N5Eru3nzmpdzT1lf2jb/c/e/uPtfJG3n7p9z9znx6xhJ7+p3o2a2t5nNl7StpMvMbEY8aXtJd5rZHZIulPQJd388roU6UtIMSfdKOs/d58bLHC3pKDObp+geq9P7jQsYmF4SJZoBAgAwWJ2eo1ZWJ51UH2/1GY88MhpyztET8y6y9TjJOdLdb4jfv1XS9+Ouz0tpZGTEZ82aFToMdFL7g37Pe6RLLw0bSxaSB6jGv73atGZ/k40HtkHWGrWLCwCAKvnpT6WPfCQar+L/vRVXlF5INDZr9hlr86yxhvT444OLraDMbLa7j3Sar9uOKj4i6VQze8jM/izp+3EZMBhVSKjSqOKBHQCAovnwh0NHkK8Xurh7pzbPV7+abywV0zGpMrNxkl7n7ptL2kzSFu6+hbv/MffoANS9+tWD3+Zhhw1+mwAAFMGOO4aOIH8nn9x62uGHDy6OCui2+d917l6pXxbN/0qiak3P+m3+18s8WWoXLwAAVVTl/321zzZ+fNRt+sorS88803yeqn32PmXd/G+mmX3GzNYzs9Vrr5QxAugXN48CAJCP1742dAT5uOqq+vhWW0XDZ58dPc/p9PfWr05dqtfU7p86IlHmkl6TbThAxbXqIv3ggwcfCwAAGGvevGpevDzggPr4lVdKK600dp6jj46Gyy03mJgqpKukyt03yDsQYKiddVboCAAAQKNbb63X6pTdk09Gw9e+Nurhr5knnoiGyQQMXem2pkpmtqmkjSX9K3V19zPzCApAC4ccIp1xRugoAAAYDltvXb17i264YfT7558fm2T97GcDC6cqurqnysz+S9J349c7JH1T0p45xgWgmVAHuY9/PMx2AQBAttZZZ/T7PfaIhlOmDD6WCum2o4r9JL1T0t/d/cOSNpe0bG5RAVI12zMnrbtuuuUHuX9+8IPBbQsAgND+9rfQEWRr3ryxZSuvHA3/8IfR83zoQwMJqWq6TapedPclkhaZ2SqSHhOdVADpPPJI6Ajao3YKADCsJk0KHUG2dtllbNkxx0TDRYtGPxT4pz8dTEwV021SNcvMVpX0I0mzJf1R0q25RQUgvOnTQ0cAAEB4kyeHjiC9hx6KhmuuWS877rj6+EYbDTScKuoqqXL3T7j70+7+A0m7SDokbgYIYNAOOSR0BAAADI8qNQW86KLm5bXP+P/+3+BiqZhuO6o408wONbON3P0hd78r78AAtDDoziqmTRvs9gAAKIJmTebKbrvt2k8/6aTBxFFB3Tb/+5mkiZK+a2YPmNkFZvaf+YUFJFStK9N2n6fXz5pXZxXJ9f7wh/lsAwCAIrvyytARZOPZZ1tPG5dIBareQVjOum3+9ztJX5f0BUk/ljQi6fAc4wLQq0WLqpeAAgCAyOTJUeKz//69Lbfttq2nve519fEvf7m/uCCp++Z/V0u6UdIBku6T9BZ35442oAjMotfSS4++4gQAALJxySXZr/Pss+v/wydM6Dx/7b6nX/2q+fSRkeYXV+fOjYaND/iVpN/+tj7+hS90jgEtje9yvrskbSlpU0n/kPS0md3k7i/mFhmG27BUQZv1V7t0yCHSGWdkH08jar4AAJD23Vd65ZVs1tXsHGfhwt7WMWGC9NhjY9c5blzr/93nnz+2bIMNoouyG2/c2/YxhnkPJ01mtpKkD0v6jKR13L20DwAeGRnxWbNmhQ4DrSQPOFU8sW/8fLX3vXzWdolnmn1W9X0PAEC38vif2Or/d6f1Ny5Xm3+NNaQnn2y9nn7OMfAvZjbb3Uc6zddt878jzeyXku6QtJekn0jaPV2IADLDgRIAgOzl+RBg99H/v594orflzaQTThidUDW6777+YkPPur0BY3lJ/yNpI3d/p7t/Oe68AkAotYMxCRUAAPkY5DOq1luv9bSLL66Pf+979fFjjx077xe/WB/fccf0caEr3fb+9y1JS0s6WJLMbC0z2yDPwABJ1U0Y3v/++vjBB2ezzuS+atc0sHZTbLN9+/GPN18fAABI70MfGlu21FLR8MU2XRV88IP18SOOGDs9eSvB175WL6/dd7XOOj2Fid512/zvvyQdLamWDi8t6ay8ggIq76yzmo/nLZlsNespcPr0wcUCAMCwOfvs7soaPfNMNKz9H3/qqfq0O++Mhp/9bDRsdlF09uzuY0Rfum3+t7ekPSU9L0nu/oiklfMKCkAOhqVHRQAA8jBxYvp1LFo0tuyAA7pfvvbMqVVXlf70p6hHws02i8pOPLH1cv/2b91vA33ptkv1l93dzcwlycyadHQPZIST//4lq/+TTfzYpwAApPP3v2e3rnXXbV5+003tH9b7m9/Ux6dMaT3fuedGXaVjYLqtqTrPzH4oaVUzO1TSVZJ+nF9YADJRu38q6WMf67wc91MBABAZ320dRA9uv715+dve1n651Vbrbv0HHxw90xID021HFSdJ+pWkCyS9XtIX3f2UPAMD0KcN2vQh4y796EeDiwUAgLK74ILs17nWWqPfL7NMNEz7gOH3vCcaLlokPf98NL755unWia50nXq7+0xJMyXJzJYys/e7exd31gEYqAcfbP2AQAAA0Js998x/G9de27rZ39e/3v16Lr107DnA9df3Hxe61ramysxWMbNjzex7ZrarRY6U9KCk9w0mRAB96/Qcq+SBl/uuAAAIY5ttWk/76lfTrXtl+pYbhE7N/36uqLnfHEkfk3SlpP0lTXX3qTnHhmFX9dqVPD8fDwUGACB7aZ4tudtu3c3344ZuC156KRrmcW8XMmPe5sTLzOa4+xvj8aUkPS5psrs/O6D4cjMyMuKzZs0KHQaaqdWYDENSkKwdGuTnbbbdULEAAFB0WfyPXGopacmS1uuobWPcOGnx4rHlH/3o2ISrmTe+Ubr77vqytW2iL2Y2291HOs3XqabqX3fLuftiSX+uQkKFAqvdYAkAAFAlnZKblVZqP183CZUkzZlTH99nn+6WQWqdkqrNzeyZ+PWspM1q42b2zCACxJC57LLQEQyHdt2qU0sFAMBoH/jA2DIzab31el9Xq+dLPfRQ7+vq5Pzzs18nmmqbVLn7Uu6+Svxa2d3HJ8ZX6XejZra/mc01syVmNqY6zcwmm9lzZvaZRNluZnafmc0zs2MS5RuY2S1mdr+Z/dLMluk3LmBo0K06AADd+/nP6+Nnn11vkjd/fu/ruvHG5uVrrNH7ulr5xS+iXgPphGpgun34b9bulrSPpOtaTP+2pCtqb+L7uU6VtLukjSUdZGYbx5NPlPRtd58i6SlJH80raKCypk0LHQEAAOXQWGv1+tf3tnzjM6qamTAhGh5wQG/rrjnwQOm44/pbFn0J0o2Iu98rSdYkezazvRR12f58ongrSfPc/cF4nnMlTTWzeyXtJOnf4/nOkPQlSaflFTtQOVzFAgCgf3/6U3brWm016amnpIULpSefzOfBw8hFqJqqpsxsRUlHS/pyw6R1Jf0t8X5+XLaGpKfdfVFDOcqO+3oAAEARmUXnKQsWZL/uJ5+sj6+xRr0XwOWXz35byFRuSZWZXWVmdzd5tXu+1ZcVNeV7rnF1Teb1NuWtYppmZrPMbNbChQs7fwggbySPAACUQ+0ZkLXe+dZZpz5txRXbL7v99t1vZ8aMsWX//d/dL48gcmv+5+4797HY1pL2M7NvSlpV0hIz+6ek2ZKS3atMkvSIoudmrWpm4+Paqlp5q5imS5ouRc+p6iM+5Ond7w4dQRhFSqxa9UgEAADGGj9eWrRIeuGF9vO16pyimV13jZ5plXxW1ac+1V98GJhCNf9z9x3cfX13X1/SdyT9t7t/T9JtkqbEPf0tI+lASZd49OTiayTtF6/iEEkXBwgdWbj88tARDJdm3apn2S4cAICqe+WVzvP0Y9GizvOgUIIkVWa2t5nNl7StpMvMrEk9Z11cC3WkpBmS7pV0nrvPjScfLekoM5un6B6r0/OLHF0zowOEoqNbdQAAstPNec+b3tT9+g4+uP9YMHDmRWp6NEAjIyM+a9as0GFUV+3A0svvK3kwGtLf5cA1/gNgvwMA0JsJE6Le+qTW/0dr/28fe6y7LtVrzj9f2mefqDkggjCz2e4+5rm6jQrV/A8VkbaGaoUVsokDAAAgb489Vh9fcUVp5szW8/aSUEnS/vuTUJUESRWK5/nnO88DAABQNC+8EHU0UbsNYscdQ0eEASGpAgAAANK44Ybm5ddfzz3mQyK3LtUBSfUH5HUzH8I69NDQEQAAUE7bbTf2fIdzm6FCTRUwzGoPMJSk6dPDxQEAQNW4Sx/5SOgoMCAkVcAw4yoaAAD5Of30KLk66KDRFzJROSRVGDyeYVUsixfTlToAAHk65xzOfSqOpArFwsn94I3jMAAAAJAGZ1PIVi9XYbhiAwAAgAogqcJgNUukVlxx8HEAAAAAGSGpQlh77BE9KA8AAAAoKZIqhHXFFaEjAAAAAFIhqUI+6HACAAAAQ4KkCvnbbbfu5iMRAwAAQAmRVCF/M2aMLSOBAgAAQEWQVCE7yy7b2/zLL59PHAAAAMAAkVQhOy+/3Nv89PoHAACACiCpwuDwsF8AAABUEEkVwnKXFizgHisAAACUFkkV8tNtorTOOvnGAQAAAOSIpArZo9YJAAAAQ4SkCgAAAABSIKnC4FGTBQAAgAohqUI2OvXst9tug4kDAAAAGDCSKgzGjBmhIwAAAAByQVIFAAAAACmQVAEAAABACuNDBzD0kvciVaEDh112CR0BAAAAMFDUVCFbV14ZOgIAAABgoEiqkF67nv/e9a7BxQEAAAAEQFKFfP32t6PfV6GJIwAAAJAQJKkys/3NbK6ZLTGzkUT5+mb2opndEb9+kJi2pZnNMbN5ZnaKWVQ9Ymarm9lMM7s/Hq4W4jMBAAAAGE6haqrulrSPpOuaTHvA3beIX4clyk+TNE3SlPhVe5rsMZKudvcpkq6O3yMEaqEAAAAwhIIkVe5+r7vf1+38ZjZR0irufpO7u6QzJe0VT54q6Yx4/IxEOQZh111DRwAAAAAEVcR7qjYws9vN7Foz2yEuW1fS/MQ88+MySVrb3RdIUjycMLhQoZkzQ0cAAAAABJXbc6rM7CpJ6zSZdLy7X9xisQWSJrv7E2a2paSLzGwTSc26l+u5rZmZTVPUhFCTJ0/udXEAAAAAGCO3pMrdd+5jmZckvRSPzzazByRtqKhmalJi1kmSHonHHzWzie6+IG4m+Fib9U+XNF2SRkZGuAEIAAAAQGqFav5nZmuZ2VLx+GsUdUjxYNys71kz2ybu9e+Dkmq1XZdIOiQePyRRXg5V6dyh3edwr78AAACAignVpfreZjZf0raSLjOzGfGkHSXdZWZ3SvqVpMPc/cl42uGSfixpnqQHJF0Rl58gaRczu1/SLvF7DEK7h/4CAAAAQ8J8SGsPRkZGfNasWaHDiNSSk7J9F8mkqmyxAwAAAB2Y2Wx3H+k0X6Ga/wEAAABA2ZBUIb1ddgkdAQAAABAMSRXSu/LK0BEAAAAAwZBUoT90UgEAAABIIqkCAAAAgFRIqgAAAAAgBZKqItl229ARAAAAAOgRSVWR3Hxz6AgAAAAA9IikCgAAAABSIKkCAAAAgBRIqgAAAAAgBZIqpOMeOgIAAAAgKJIqAAAAAEiBpAoAAAAAUiCpAgAAAIAUSKoAAAAAIAWSKgAAAABIgaSqCLbZJnQEAAAAAPpEUlUEN90UOoLemIWOAAAAACgMkioAAAAASIGkCgAAAABSIKkCAAAAgBRIqgAAAAAgBZIqAAAAAEiBpAoAAAAAUiCpAgAAAIAUSKrQP/fQEQAAAADBkVQVDQ/WBQAAAEqFpAoAAAAAUiCpAgAAAIAUSKoAAAAAIAWSKgAAAABIIUhSZWb7m9lcM1tiZiMN0zYzs5vi6XPMbLm4fMv4/TwzO8Us6tHBzFY3s5lmdn88XC3EZwIAAAAwnELVVN0taR9J1yULzWy8pLMkHebum0h6u6RX4smnSZomaUr82i0uP0bS1e4+RdLV8XvkZeedQ0cAAAAAFEqQpMrd73X3+5pM2lXSXe5+ZzzfE+6+2MwmSlrF3W9yd5d0pqS94mWmSjojHj8jUY48XH116AgAAACAQinaPVUbSnIzm2FmfzSzz8Xl60qan5hvflwmSWu7+wJJiocTBhYtAAAAgKE3Pq8Vm9lVktZpMul4d7+4TTzbS3qLpBckXW1msyU902Re7yOmaYqaEGry5Mm9Lg4AAAAAY+SWVLl7PzffzJd0rbs/LklmdrmkNyu6z2pSYr5Jkh6Jxx81s4nuviBuJvhYm5imS5ouSSMjIz0nZZUU9fchbbGFdPvtYWMBAAAASqhozf9mSNrMzFaIO614m6R74mZ9z5rZNnGvfx+UVKvtukTSIfH4IYnycvHAOd4dd4TdPgAAAFBSobpU39vM5kvaVtJlZjZDktz9KUn/I+k2SXdI+qO7XxYvdrikH0uaJ+kBSVfE5SdI2sXM7pe0S/we/ajVWgEAAADomnnoGpJARkZGfNasWaHDGK2W1AzyO2mWSLXbfnL+If3tAAAAYDiY2Wx3H+k0X9Ga/6EIzKR3vrP9PONzux0PAAAAKBWSKkQaa51+97souTKTttxy7PyvvDK2DAAAABhCJFVllnXzu1br++Mfs90OAAAAUCG04SqrLO5t6nQ/FR1XAAAAAB1RU4XW6IgCAAAA6IikCt2h1goAAABoiqSqiLbdNnQEAAAAALpEUlVEN9/cfjq1RgAAAEBhkFSh/b1TdFwBAAAAtEVSVQU77BA6AgAAAGBokVRVwQ03hI4AAAAAGFokVWWTVRM8mvIBAAAAmSCpQmc8rwoAAABoiaQKAAAAAFIgqSqr7bYLHQEAAAAAkVSV1w030CwPAAAAKACSqjLJo3OJt7+9u/mSCRzJHAAAAPAvJFVFss022a3LrLsk7Jprel/34sW9LwMAAABU1PjQASDhpptaJ0K91FLl1V06NVQAAADAGCRVRdUuMWqW3JiR9AAAAAAB0PyvbN761t7mf9vbxpbx4F8AAAAgM9RUFc1b3iLddtvosldekcb3+VVdd136mAAAAAC0RFJVNLfeGjoCAAAAAD2g+V/V0LQPAAAAGCiSKgAAAABIgaSq7NL0+EdvgQAAAEBqJFXDgCaBAAAAQG5IqqqqVS0UCRYAAACQKZIqAAAAAEiBpKpKGmuhdtwxTBwAAADAECGpqrJrrx39Ppl00UkFAAAAkAmSqmHBvVQAAABALoIkVWa2v5nNNbMlZjaSKH+/md2ReC0xsy3iaVua2Rwzm2dmp5hFWYKZrW5mM83s/ni4WojPVCg77BA6AgAAAGBohKqpulvSPpKuSxa6+9nuvoW7byHpYEkPufsd8eTTJE2TNCV+7RaXHyPpanefIunq+P1wu+669tNp+gcAAABkJkhS5e73uvt9HWY7SNIvJMnMJkpaxd1vcneXdKakveL5pko6Ix4/I1E+PEiSAAAAgGDGhw6gjQMUJUyStK6k+Ylp8+MySVrb3RdIkrsvMLMJgwuxBNxH30/1treFiwUAAACooNySKjO7StI6TSYd7+4Xd1h2a0kvuPvdtaIms/VcPWNm0xQ1IdTkyZN7Xbwafv/70BEAAAAAlZJbUuXuO6dY/EDFTf9i8yVNSryfJOmRePxRM5sY11JNlPRYm5imS5ouSSMjI7SZAwAAAJBa4bpUN7NxkvaXdG6tLG7e96yZbRP3+vdBSbXarkskHRKPH5IoH07t7q966aXBxQEAAAAMiVBdqu9tZvMlbSvpMjObkZi8o6T57v5gw2KHS/qxpHmSHpB0RVx+gqRdzOx+SbvE74ePe+uEqjZtmWUGGxMAAAAwBMyHtOe4kZERnzVrVugwAAAAABSUmc1295FO8xWu+R8AAAAAlAlJFQAAAACkQFIFAAAAACmQVAEAAABACiRVAAAAAJACSRUAAAAApEBSBQAAAAApkFQBAAAAQAokVQAAAACQAkkVAAAAAKRAUgUAAAAAKZBUAQAAAEAK5u6hYwjCzBZK+kvoOGJrSno8dBBDgP2cP/Zx/tjH+WMf5499PBjs5/yxj/MXeh+/2t3X6jTT0CZVRWJms9x9JHQcVcd+zh/7OH/s4/yxj/PHPh4M9nP+2Mf5K8s+pvkfAAAAAKRAUgUAAAAAKZBUFcP00AEMCfZz/tjH+WMf5499nD/28WCwn/PHPs5fKfYx91QBAAAAQArUVAEAAABACiRVgZnZbmZ2n5nNM7NjQsdTBWa2npldY2b3mtlcM/vPuPxLZvawmd0Rv/YIHWuZmdlDZjYn3pez4rLVzWymmd0fD1cLHWdZmdnrE7/VO8zsGTP7FL/j9MzsJ2b2mJndnShr+tu1yCnxMfouM3tzuMjLo8U+/paZ/V+8Hy80s1Xj8vXN7MXEb/oH4SIvjxb7uOXxwcyOjX/H95nZu8JEXS4t9vEvE/v3ITO7Iy7nd9yHNudspTsm0/wvIDNbStKfJO0iab6k2yQd5O73BA2s5MxsoqSJ7v5HM1tZ0mxJe0l6n6Tn3P2koAFWhJk9JGnE3R9PlH1T0pPufkJ8kWA1dz86VIxVER8rHpa0taQPi99xKma2o6TnJJ3p7pvGZU1/u/FJ6Scl7aFo//+vu28dKvayaLGPd5X0O3dfZGYnSlK8j9eX9JvafOhOi338JTU5PpjZxpJ+IWkrSf8m6SpJG7r74oEGXTLN9nHD9JMl/cPdv8LvuD9tztk+pJIdk6mpCmsrSfPc/UF3f1nSuZKmBo6p9Nx9gbv/MR5/VtK9ktYNG9XQmCrpjHj8DEUHRqT3TkkPuHtRHlheau5+naQnG4pb/XanKjqhcne/WdKq8UkA2mi2j939SndfFL+9WdKkgQdWIS1+x61MlXSuu7/k7n+WNE/ROQjaaLePzcwUXaz9xUCDqpg252ylOyaTVIW1rqS/Jd7PFyf/mYqvHL1J0i1x0ZFxdfFPaJqWmku60sxmm9m0uGxtd18gRQdKSROCRVctB2r0P25+x9lr9dvlOJ2Pj0i6IvF+AzO73cyuNbMdQgVVEc2OD/yOs7eDpEfd/f5EGb/jFBrO2Up3TCapCsualNEeM5P1LI8AAASfSURBVCNmtpKkCyR9yt2fkXSapNdK2kLSAkknBwyvCrZz9zdL2l3SEXEzCWTMzJaRtKek8+MifseDxXE6Y2Z2vKRFks6OixZImuzub5J0lKRzzGyVUPGVXKvjA7/j7B2k0Re7+B2n0OScreWsTcoK8VsmqQprvqT1Eu8nSXokUCyVYmZLK/rjPNvdfy1J7v6ouy929yWSfiSaPqTi7o/Ew8ckXahofz5aq4aPh4+Fi7Aydpf0R3d/VOJ3nKNWv12O0xkys0MkvUfS+z2+qTtukvZEPD5b0gOSNgwXZXm1OT7wO86QmY2XtI+kX9bK+B33r9k5m0p4TCapCus2SVPMbIP4avSBki4JHFPpxe2cT5d0r7v/T6I82eZ2b0l3Ny6L7pjZivENpTKzFSXtqmh/XiLpkHi2QyRdHCbCShl1NZTfcW5a/XYvkfTBuMepbRTdlL4gRIBlZ2a7STpa0p7u/kKifK24MxaZ2WskTZH0YJgoy63N8eESSQea2bJmtoGifXzroOOrkJ0l/Z+7z68V8DvuT6tzNpXwmDw+dADDLO4B6UhJMyQtJekn7j43cFhVsJ2kgyXNqXV1Kuk4SQeZ2RaKqokfkvTxMOFVwtqSLoyOhRov6Rx3/62Z3SbpPDP7qKS/Sto/YIylZ2YrKOodNPlb/Sa/43TM7BeS3i5pTTObL+m/JJ2g5r/dyxX1MjVP0guKel9EBy328bGSlpU0Mz523Ozuh0naUdJXzGyRpMWSDnP3bjtgGFot9vHbmx0f3H2umZ0n6R5FTS+PoOe/zprtY3c/XWPvc5X4Hfer1Tlb6Y7JdKkOAAAAACnQ/A8AAAAAUiCpAgAAAIAUSKoAAAAAIAWSKgAAAABIgaQKAAAAAFIgqQIAVIKZLTazOxKvYzJc9/pmxjPBAABN8ZwqAEBVvOjuW4QOAgAwfKipAgBUmpk9ZGYnmtmt8et1cfmrzexqM7srHk6Oy9c2swvN7M749dZ4VUuZ2Y/MbK6ZXWlmy8fz/4eZ3ROv59xAHxMAEBBJFQCgKpZvaP53QGLaM+6+laTvSfpOXPY9SWe6+2aSzpZ0Slx+iqRr3X1zSW+WNDcunyLpVHffRNLTkvaNy4+R9KZ4PYfl9eEAAMVl7h46BgAAUjOz59x9pSblD0nayd0fNLOlJf3d3dcws8clTXT3V+LyBe6+ppktlDTJ3V9KrGN9STPdfUr8/mhJS7v718zst5Kek3SRpIvc/bmcPyoAoGCoqQIADANvMd5qnmZeSowvVv2+5HdLOlXSlpJmmxn3KwPAkCGpAgAMgwMSw5vi8T9IOjAef7+kG+LxqyUdLklmtpSZrdJqpWY2TtJ67n6NpM9JWlXSmNoyAEC1cTUNAFAVy5vZHYn3v3X3Wrfqy5rZLYouJh4Ul/2HpJ+Y2WclLZT04bj8PyVNN7OPKqqROlzSghbbXErSWWb2Kkkm6dvu/nRmnwgAUArcUwUAqLT4nqoRd388dCwAgGqi+R8AAAAApEBNFQAAAACkQE0VAAAAAKRAUgUAAAAAKZBUAQAAAEAKJFUAAAAAkAJJFQAAAACkQFIFAAAAACn8f+hEO/03MTALAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "env = gym.make('Pendulum-v0')\n",
    "agent = Agent(3, 'pendulum')\n",
    "agent.train()\n",
    "optimiser = optim.Adam(agent.parameters(), lr=1e-3)\n",
    "entropy_loss_weight = 0.0001\n",
    "total_rewards = []\n",
    "\n",
    "plt.figure(figsize=(14, 6))\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Reward')\n",
    "\n",
    "for _ in range(epochs):\n",
    "    optimiser.zero_grad()\n",
    "    total_reward = 0\n",
    "    for _ in range(rollouts):\n",
    "        obs, done = obs_to_tensor(env.reset()), False\n",
    "        values, log_probs_action, rewards, entropies = [], [], [], []\n",
    "        while not done:\n",
    "            policy, value = agent(obs)\n",
    "            action = policy.sample()\n",
    "            obs, reward, done, _ = env.step([action.item()])\n",
    "            obs = obs_to_tensor(obs)\n",
    "            total_reward += reward\n",
    "\n",
    "            rewards.append(reward)\n",
    "            values.append(value)\n",
    "            log_probs_action.append(policy.log_prob(action))\n",
    "            entropies.append(policy.entropy())\n",
    "\n",
    "        ep_return, gae = torch.zeros(1), torch.zeros(1)\n",
    "        values.append(ep_return)\n",
    "        trajectory_length = len(rewards)\n",
    "        loss = 0\n",
    "        for i in reversed(range(trajectory_length)):\n",
    "            ep_return = rewards[i] + discount * ep_return\n",
    "            advantage = ep_return - values[i]\n",
    "            loss += value_loss_weight * advantage ** 2\n",
    "            td_error = rewards[i] + discount * values[i + 1].item() - values[i].item()\n",
    "            gae = gae * discount * trace_decay + td_error\n",
    "            loss -= log_probs_action[i] * gae\n",
    "            loss -= entropy_loss_weight * entropies[i]\n",
    "        loss.backward()\n",
    "    optimiser.step()\n",
    "\n",
    "    total_rewards.append(total_reward / rollouts)\n",
    "    plot()\n",
    "\n",
    "clear_output(wait=True)\n",
    "display('Average final reward: %.2f' % total_rewards[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQsAAAD8CAYAAABgtYFHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAABLVJREFUeJzt201O3FgYQFEbwTx76SyABYSltdR7QOoNhGV0ZplnIUi4Bw0IQui6lSpjl+ucGb96g+Lyvvdc4zRNA8AuF0svADgNYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkl0sv4JHHSGF+4yE/bGcBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGIBJGKxUt9uboZvNzdLLwOejdM0Lb2GYRiGVSxiDd4LxB9fv37wStig8aAfFov1KDsJ0eAAB8XCGLISdeQwmrAUsThBgsESxOJECQYfTSxOmBsTPpJYbIBg8BHEYiUOveUQDObm6nSFDv3Dd73KO1yd8ppdBnOws1gxOwyOzBOcW3dINASDF4whvM/1KsdiZ3FCjCUcyM7iXLheZUlicWIEg6UYQ06Yg0/2ZAxhfw4+2ZedxQY4+CTynAX/MZawgzGEwxlL2MXOYoOMJbzDzoLXXK8yB7HYKMHg2IwhZ8DBJ4/chrCbcwwGZxYUxx5LxnEcxvGg1x4nRizOyDGDMU3T8LQrFY3zYAw5U3OMJeM4Dit5PfFrxhD2N8dtyTRNdhkbJhZnbK5gsE1icebmCIbdxTY5s+CZ5zE2z3MWHM9TMD7f3Q3/fPny5uuf7+5effzye14G4+LiYnh4eJhplfwmB5wcz7F2CCv5J8QRiQVv/PXp0zAMw/D3jx8Lr4Q1EQveuL29Ha6vr4c/v3//7d/hkHN7nFmwUzn4dMB5EhxwMo/7+/vh6urq+eP3ovFzKDzJuVpiwTx+9Uf/czDsKE6KWDCffXcJdhWr5uqU+ezzfg+h2DaxYKddb0V/+rxQbJsxhL29jMZKXj80B40hl8daBedDIM6TMQRIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJIxAJILpdewKNx6QUA/8/OAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkjEAkj+Bc6qBbbjtLgvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.axis('off')\n",
    "\n",
    "agent.eval()\n",
    "obs, done = obs_to_tensor(env.reset()), False\n",
    "view = plt.imshow(env.render(mode='rgb_array'))\n",
    "while not done:\n",
    "    view.set_data(env.render(mode='rgb_array'))\n",
    "    display(plt.gcf())\n",
    "    clear_output(wait=True)\n",
    "    with torch.no_grad():\n",
    "        obs, _, done, _ = env.step([agent(obs)[0].sample().item()])\n",
    "    obs = obs_to_tensor(obs)\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PPO\n",
    "\n",
    "### Training and Testing\n",
    "\n",
    "First-order gradients point in the direction of steepest ascent/descent, but the accuracy of this decreases further away from the point of evaluation. A related problem is that a large step size can cause the policy to hop around the optimisation landscape or even get worse. To prevent this, we can restrict how much the policy can change within an update. Trust region policy optimisation (TRPO) defines a hard constraint on this change (the trust region), and uses an approximate second-order method with line search to make sure the policy update never outsteps the trust region. However, this is very expensive, so PPO utilises a first-order method with a soft constraint on the change in the policy.\n",
    "\n",
    "Both TRPO and PPO perform multiple updates using data collected from rollouts, so use importance sampling to correct for the mismatch as the current policy deviates from the original policy used to collect the data (initially the probability ratio will be 1). However, as the probability ratios increase or diverge within the inner optimisation steps, the objective function becomes more innacurate. Therefore, PPO implements a clipped objective function, which makes the updates conservative and - ideally - more stable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Average final reward: -328.19'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1UAAAF3CAYAAABNBGHSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm4HFWd//HPNyuELcGENcTAGGRAGcRmExWGJaxjgGEdRpFREAHFcZCAzA/EhVVhBBGNgMCgQBDBKEvYVHSU5V72CAyBAQlE9sUEkpDc7++Pqrar+1Z1V3d1d/Xyfj1PP1V16tTpc9vmej85p06ZuwsAAAAA0JgReXcAAAAAALoZoQoAAAAAMiBUAQAAAEAGhCoAAAAAyIBQBQAAAAAZEKoAAAAAIANCFQAAAABkQKgCAAAAgAwIVQAAAACQAaEKAAAAADIYlXcH8jJx4kSfOnVq3t0AAAAA0KEGBwdfcfdJter1baiaOnWqBgYG8u4GAAAAgA5lZs+mqcf0PwAAAADIgFAFAAAAABkQqgAAAAAgA0IVAAAAAGRAqAIAAACADAhVAAAAAJABoQoAAAAAMiBUAQAAAEAGhCoAAAAAyIBQBQAAAAAZEKoAAAAAIANCFQAAAID8rVghXXONtGxZ3j2pG6EKAAAAQP5+/nPp4IOlG2/Muyd1I1QBAAAAyJe7dMYZ0sYbS5/4RN69qduovDsAAAAAoM/deqv0wAPSxRdLI0fm3Zu6MVIFAAAAIF9nnimtv770yU/m3ZOGMFIFAAAAID933y395jfSuedKY8bk3ZuGMFIFAAAAoHXcg+2KFdLDDwcLUSxfXjp/xhnSmmtKRxyRT/+agJEqAAAAAK2xfLk0dqw0NFRevtpqwVS/Qw+V5syRTj1VWnXVfPrYBIxUAQAAAGiNf/u3IFCNHCmts4606abB6513pO9/X9p+e2n0aOkLX8i7p5kwUgUAAACgNa6+Otjee6+05Zal8uXLg3uoLrwwWEb9Pe/Jp39NYl6c49hnCoWCDwwM5N0NAAAAoDfdd5+09dbBSNSyZcn13CWz9vWrDmY26O6FWvWY/gcAAACg+Q44INgeemj1eh0aqOpBqAIAAADQfM8+G2x/9KN8+9EGHRuqzOx4M3Mzmxgem5mdb2bzzexhM9syUvcwM3syfB2WX68BAAAA/G3hifXWk0b1/jIOHfkTmtkGknaV9OdI8R6SpoWvbSRdJGkbM1tT0qmSCpJc0qCZzXH319vbawAAAACSSqNTV12Vbz/apFNHqs6TdIKCkFQ0Q9IVHrhb0ngzW1fSbpJuc/fXwiB1m6Td295jAAAAANK8edLSpcEy6h//eN69aYuOC1Vm9glJz7v7QxWn1pf0XOR4QViWVA4AAACg3XbZJdjutVe+/WijXKb/mdntktaJOXWypK9Kmh53WUyZVymPe98jJR0pSVOmTEnVVwAAAAB1ePHFYPuzn+XbjzbKJVS5+y5x5Wb2QUkbSnrIgqUVJ0u638y2VjACtUGk+mRJL4TlO1aU/ybhfWdJmiUFz6nK8jMAAAAAqPDuu6XnTo0enXdv2qajpv+5+yPuvpa7T3X3qQoC05bu/hdJcyR9KlwFcFtJb7r7QklzJU03swlmNkHBKNfcvH4GAAAAoOcdc4y0225BgIo67bRgu8Ya7e9Tjjpy9b8EN0naU9J8SW9LOlyS3P01M/uGpPvCel9399fy6SIAAADQB77//WD7yivSpEml8osvDrb77tv+PuXIvDJd9olCoeADAwN5dwMAAADoLjvvLN15Z7C/1VbSvfeWzo0YEYxeLV4sjRuXT/+ayMwG3b1Qq15HTf8DAAAA0OGKgUqSBgfLzxUHbHogUNWDUAUAAAAgndmzy4+Hhkr7xQf+rrJK+/rTIQhVAAAAANL5138NtltsIa28crB/6KHB9owzgu0OO7S/XzkjVAEAAACobenSYMl0SXrgAen444P9a68Ntn/+c7AtLlbRRwhVAAAAAGrbZJNgu9JKwfbrXw+2xaC1YkWwXXfd9varAxCqAAAAANT2zDPB9rbbSmWjwic0HX54sC0Grj5DqAIAAABQ3dFHl/Y/+tHS/t57B9vLLgu2W2zRti51EkIVAAAAgOp+8INgO316efk115QfX3RRe/rTYQhVAAAAAKorPn9q7tzy8jFjJLPSMSNVAAAAAFBhm22C7YiE6LD55sF25Mj29KcDEaoAAAAAJLv33mA7c2b8+TvvDEar/vmf29enDjMq7w4AAAAA6FBvvFHaP/30+DprrikNDbWnPx2KkSoAAAAA8TbdNNiOHZtvPzocoQoAAABAvIULg23lKn8oQ6gCAAAAMFzxXipJmjEjv350AUIVAAAA0KteeCFYRGLq1Pqv3WmnYDthQlO71IsIVQAAAECv2njjYPvss/Vd98gj0uLFwf799ze3Tz2IUAUAAAD0qmIwqscll5SePWUmvfe9ze1TDyJUAQAAAP3gy1+uXedzn5M++9lgf8SIYKl0s9b2qwcQqgAAAIBe9IEPlB9/97vV6++xhzRrVrA/bpy0YkVr+tWDCFUAAABAJxo9Wtpii8avnzcv2BYXmqj1gN5bbgm2kyc3Nm2wjxGqAAAAgE4zbZq0fLn00EPZ23rttdp1ootRPPdc9vfsM4QqAAAAoNPMn5++7p//LI0cKd11V6ls5sz4uqecEl++zTbBdsyY9O+LvyFUAQAAAHmZNEm6+urystVXLz+uNgXwuuuC1fmGhqQddiiVn312sB01qnz7rW/Ft7N8ebBtxshYHyJUAQAAAHnYckvplVekQw6RTj+9VP7Xv5bXSwo6X/qStP/+5WWVK/X98Y/B9txzg23cfVWHH17a32ST2v3GMIQqAAAAIA8PPFDaP/lk6bzzgsUpin7wg+Rrt9qqfDW/YniSpJVWKu0XCsH2C19Ibuuyy4LtppvW7DLiEaoAAACATvDlL5em4e25Z/DMqCQDA6V9d2nbbaW99w6Oly6t/j7f/nZp/+23S/vF1QJRN0IVAAAAkCf34WU33lh+fOWVpf1p0+Kv/eUvpZVXLh3/x3+Ut1G8r+qkk0plG25YX18Ri1AFAAAAtNuWW5YfR8PR0UcPr3/YYaX94sqA48cPrxcdeTrjjPJzxePiaNjixdJLL8XXRV3M45JxHygUCj4QHTYFAAAA2iW6oET07/G335bGjSsd//3fS48/Xl6veG0jf8dXLmQR1wf8jZkNunuhVj1GqgAAAIC8XHRR+XE0UEnSY4+VH0+d2vw+nHde89vsM4QqAAAAIC9HHZW+7uuvS88+G+xPmtTY+510kjR2rHTttcHolHuwNDsyIVQBAAAA7fTBD2a/rngvVL1OP11asmT4862QCaEKAAAAaKdHH62vfnHVvuefb35f0BSEKgAAACAPlfdTJbnllvLj9dZrfl+QCaEKAAAAyEPa+6l23rn8mBGrjtORocrMvmBmT5jZPDM7O1J+kpnND8/tFinfPSybb2Yn5tNrAAAAQMGy5XGv732v8fup0NFG5d2BSmb2j5JmSNrc3Zea2Vph+aaSDpa0maT1JN1uZhuHl10oaVdJCyTdZ2Zz3P1P7e89AAAA+tqLLyaf+8IXGm93xAhpaEjaaKPG20DLdFyokvR5SWe6+1JJcvfi0iYzJF0dlv+fmc2XtHV4br67Py1JZnZ1WJdQBQAAgOqWLZPGjGlee+uvX9ovPlD35ZeltdYqr/fjH9fX7pIlwWjXqE788x2dOP1vY0kfM7N7zOy3ZrZVWL6+pOci9RaEZUnlAAAAQLIJE4JnNplVr3fUUaUpfAMD1euuWBFsR48ulU2aFASs6DLon/50fX0dPZpA1cFy+V/GzG6XtE7MqZMV9GmCpG0lbSVptpltJCnu2+6KD4ae8L5HSjpSkqZMmVJ/xwEAANA73nijtG8WPBA3+vymlVcORoiittqqNAJVzbJlw8uK4Qo9J5dQ5e67JJ0zs89L+rm7u6R7zWxI0kQFI1AbRKpOlvRCuJ9UXvm+syTNkqRCocA3GgAAACUHHJDt+h/+sDn9QNfpxOl/N0jaSZLChSjGSHpF0hxJB5vZWDPbUNI0SfdKuk/SNDPb0MzGKFjMYk4uPQcAAEB3mDmztF9t9GjEiOB8tM7EifF10y6Rjp7TiRMzL5V0qZk9KmmZpMPCUat5ZjZbwQIUyyUd4+4rJMnMjpU0V9JISZe6+7x8ug4AAICucPbZ5cfu5fdWPf988kN2X321ettbbJGtb+g65n06r7NQKPhArRsNAQAA0JuKAWqHHaTf/Kb+6wYHpS23jD/Xp39f9yIzG3T3Qq16nTj9DwAAAGiPegJV1Ic/XH5cqPl3N3oYoQoAAAD9pdYS6tUcc0x8+eBg422i6xGqAAAAgLS+973SftyCFeec076+oGMQqgAAANC77rsvGJk644zh57Le+/Tqq0Hbf/lLqez447O1ia5EqAIAAEDv2nrrYPvVrwYP89199+xtVoaxddfN3ia6GqEKAAAA/WHJEmnu3Oa05S5ttFF52ejRzWkbXYdQBQAAgP7zuc9lb+Opp0oPBh4YkJYty94muhKhCgAAAL3p2WdL++7S+99fOr7ooua+V+US6+gro/LuAAAAANASG25Yfvz440G4Wro027LqQAVGqgAAAJCP6dODcPPpT7em/bjV/cyklVZqzfuhbxGqAAAAkI/bbgu2l1/e2vfZa6/Wto++R6gCAABA65x3Xrqpdqed1ro+/OpXrWsbkGSe9aFnXapQKPjAwEDe3QAAAOht0UBV+XdnZdiq5+/S4rXVrklTB6jCzAbdvVCrHiNVAAAAaI1qI1TbbDO87Kab6m836T3GjEnXFtAEhCoAAADUZha8Zs9OX7+ae+8dXpbm3qe4ds2kk08uL3v33dptAU1CqAIAAEB6Bx1Uu87HPpa+vREjpK99rXT85JPx9dzLA9VZZ0n77FM6Pv106cADh1/3oQ+l7wvQIO6pAgAAQG313P9UeR9V3H1Vlfc7Vbv3qvL8WWdJJ5yQ/H5x7QMN4J4qAAAANMcaawwvS5reVyscJdlll+Rzl15a2p85szxQVb7Peuulf0+gSQhVAAAAqO6tt+LLv/OddNc//nj5cVwgKz6zKu78Zz5T2j/zzOrvtXChtNpq6foFNAmhCgAAAOlFR4WOP7783He/G1/v/e8v7T/xRHLbtUaZKt8vqV+LFlVvB2gyQhUAAEC/mj493YN5i1ZaKdhGA8zUqaX9L32pdhubbFLaP/TQ8nPPP1/aL/Yr2r9zzknVzb9Zf/366gMNIlQBAAD0o299qzTlLm2weued4WXPPju8rNr9UVFXXjm8bPTo+LpJ5VGV93AtWJCuH0BGhCoAAIB+9J//me36aICZOFEaNap0HL0/qqjyvqoky5aV9qNhL1peTdyiGkCLEaoAAAB6zQ03VD9fz5S/NF59VVqxonqd6H1VrfTGG8F23Lj2vB8gQhUAAEBvMZP23TfYPvBA/Pmib36zdnvV7mOKWzK9Gc+Fqmyj3jbdpcWLs/cDSIlQBQAA0CvWWaf8eMstpdNOC/aPPro8UE2eLJ18cuk4aSSp8plQzcJDedFDzPv0C10oFHxgYCDvbgAAADRP2ml9I0aUpuvVelhvmof5FuustlryM60q66b5G3T6dOnWW2vXA1rEzAbdvVCrHiNVAAAAvaAy/CSFFvfa9z/Va+LEYFsrUElBoFt77XTtEqjQJUbVrgIAAICOFg1URxxR2nevPdJUWSfJv/978rmXX659fVGzAx3QAQhVAAAAvWTWrPLj4qhV1hX/zj032/VAD2P6HwAAQDer556netoCkBqhCgAAoFMtX553DwCkQKgCAABoRPEhs62w3nrBqNHo0dJKK6W7JsuKznvsEV++8sqNtwn0EUIVAABAvcykCROaP13uO98J2ly4sFS2dGn1fjTDTTfFly9Z0pz2gR5HqAIAAP3rmmuk972vvmsqg0ytYPPOO9IvflG73W23lY4/vr6+tMK//EvePQC6TseFKjPbwszuNrMHzWzAzLYOy83Mzjez+Wb2sJltGbnmMDN7Mnwdll/vAQBA1/j1r6WDD5aeeio5GC1eXH6cVC+p/H3vk8aNk/bZR9p//+r9ueee0v4FF5RP54trf3CwtJ9l6l+lq64aXrbKKs1rH+hBnbik+tmSTnP3m81sz/B4R0l7SJoWvraRdJGkbcxsTUmnSipIckmDZjbH3V/Po/MAAKBL7LRT9fPVRqA220x69NHyOmbl4ebAA4PAVnTddeneK21AKhTS1WuGRYva915AF+q4kSoFwWj1cH8NSS+E+zMkXeGBuyWNN7N1Je0m6TZ3fy0MUrdJ2r3dnQYAAF0kLjBVBqQkM2YEgUoaHoDMgtdxx0nXXpvufaMjVJXSBKyxY2vXSeOhh0r7e+/dnDaBPtGJoepLks4xs+ckfVvSSWH5+pKei9RbEJYllQMAAAz361+XH0dXviuGoqIpU0oPz/3Nb4L7o264ofz6uOBz/vnVz0dtu226utF+7bNPab9Zi0lsvnlp/8YbpbPOak67QB/IZfqfmd0uaZ2YUydL2lnSv7v7dWZ2oKRLJO0iKe6fjLxKedz7HinpSEmaMmVKAz0HAABdLzrtrxhikkaunn22dLzDDsltJrVTLHcvnYtOE4yOMh16aO2+S9LAQLqFLxoR7eeJJ7bmPYAelMtIlbvv4u4fiHn9QtJhkn4eVr1W0tbh/gJJG0SamaxgamBSedz7znL3grsXJk2a1MwfCQAAdINo6DnllNJ+3AjR0FD97btLv/99sP+731Xvh5m0bFmp7Mork9ssGhiQttqqdHzfffX3sZZmLnoB9IlOnP73gqTiPwXtJOnJcH+OpE+FqwBuK+lNd18oaa6k6WY2wcwmSJoelgEAgEqNBIVeddpp5cfRaYFZgsX22wfXf/Sj5eXV2qz2LKqoykDVqsUqHn64Ne0CPaoTV/87QtJ3zWyUpCUKp+tJuknSnpLmS3pb0uGS5O6vmdk3JBX/qebr7v5ae7sMAECXGDmytN8JIxLFkaNm9qXY5lprSS++OLw86f123LH1n8n48dIbb1TvRxqtDFSS9MEPSj/5STAlkYAF1GTeCb9Qc1AoFHxgYCDvbgAA0F6NLN3dKkn3HzWz3aRnPeX5s0fvW6pH8ZqLLpKOOqq5fQIQy8wG3b3mv2B04kgVAADodUkLQzQ77BTb7JRAJTUWqCRpxQppRCfeuQGA/zIBAOgXEyfm3YPACSckn2s0cFS7PmubnYJABXQs/usEAKBfvPpq+XFeYeOcc0r7xWdARbWyX3mPUgHoSYQqAAAQ7513mr9aYDQw3Xhjab/ZwSourAFAixCqAADodkNDpecepTFuXO06118f1Bs5stT2tGnZ+llpzz3Lj7OGoLif//bbS/ssJw+gRQhVAAB0u+gy6cUAVC1gLV5cu8399hteNn9+thGkNItFJK3W16idd5aWLCkFTwBoAUIVAAC9as01S/tJo1O1Fnb44hfTXdNM0YfmRkNi2tG4ysA2diyBCkBLEaoAAOgVlfcRvf56af+dd9K1URk+vvvd5iwmUc+S5r/7Xe22zKRbb228PwDQRIQqAAC6WVyYeO97a1937LHDyypHpSrDj7u0zjrl792qxSDStLvbbvXdSwYALUKoAgCg1zzzTGm/MnBce22wveCCUtl11wX1omVJoWbhQmnvvUvHaZ6dFH0+Vj0hrDhCVvlaeeXq1wBAmxGqAADodFddVd89RdXsv3/tsuuuq97GL39Z33tWPh8rq7ffZsl0AB1lVN4dAAAACeoJUJUBY/Zs6cADg/00o0lJ7VSrV+xftWmAN99c2r/hhvT9SItgBaADmPfpL6NCoeADAwN5dwMAgHijRkkrViSfj4aa4nGluFBWbcGJev8mSHNtlvYBIGdmNujuhVr1mP4HAEA7uUvLl9euFw1UrZzqtv76wfTCRtqv55lSa6xRf/sA0CUIVQAAZFXP/U4jRkijR0sXXli9vaJocEkKMVdfHd9OZVAqLlIRtWCBdPDByX3JItrHN95ozXsAQAcgVAEAul8x0HznO9nbSjOK1KhoyIhb0rxRBx2Url7cIhVZ1TNaBQA9ilAFAOgdxx/f+LXFYDZ6dH3hoLJu0rVp26x1D9JVV6VrJy/RxSuKuJcKQI8jVAEAulvaUFNPG5J0ySWN9SfODjuke9+f/rR2W/VO1WtHoKm2+AUA9AFCFQCg96T9o/7QQ5Prfvazta+/7LLS/o9/nPz+d91V2q8Wcg49NF296GjV0FDVLv6trVaHq6T2GaUC0AcIVQCA7lVtilmaYFU5MuQuXXxx+jYOP7y0/+lPl5+7/PLhi1dcccXwvsZNl1tllervGx2t6qRRIQIUgD5FqAIA1O/b306/2l019ayal0Y9wSp6btas0rWf+UxyH5P6Onbs8PevDFmS9MlP1u6LJC1alNzvonaMPjUi2qdqz9kCgB5CqAIA1O8rX2l+m/UGqzTLjie1W1l2xBHlx7XCSmW4WrKktD9t2vD6cQGol6fLFX/eEfyZAaA/8NsOANA5LrigOe1UC1YjR1avGy3/y1+C6YAbbZT+vf/3f8vbqCck9UKgAoA+VDVUmdkjZvZw0qtdnQQAdJDKZ0E1856eL34xXb00y3XHBSuz8oUdaoWYtdcOpgM+9VQpIF16afX3KZalCUjFOu9/P4EKALpYrZGqvSX9k6Rbwteh4esmST9rbdcAALk6++wghJxxRnl5lmdBRUWD0fnnl5e36h6rtOeqOfzw4NqhIWnx4sbaqOzH449nbwcAkBvzFP+nYmb/4+7b1yrrJoVCwQcGBvLuBgB0rqTRoLjA00hAqWy/WpCKtj96tLR8ebB/003SHnuke69DDkn3HCgAAEJmNujuhVr1RqVsbxUz+6i7/z5s/COSaqz3CgDoWnEP1HUvn/oXDULF81lUC1bR9ouBSkoXqIptAwDQImlD1b9J+rGZrSHJJb0ZlgEAek21EaNmTf2Lqmflvk02Ke1feGHz+wIAQANqhiozGyHpfe7+D2a2uoIpg2+2vmsAgLarDDGVo1HNEr2Hqppq73/00c3rDwAAGdRcUt3dhyQdG+6/RaACgB618cblx2kWeWh0Wt1xxzV2XdEBB2S7HgCAJkr7nKrbzOx4M9vAzNYsvlraMwBAez35ZGm/2pS8JEkP2Y2+GhH3/rNnN9YWAAAtUM89VZJ0TKTMJdXxNEQAQMeKBp6zzmpeW7WstVa6etFpgJMm1d8nAABaKNVIlbtvGPMiUAFALzrhhOFl0dGiZcvKz0VXBKwVqCrPv/hi+n4NDUlLlkgvvZT+GgAA2iDVc6okycw+IGlTSSsVy9z9ihb1q+V4ThUAhJKeR1Wp1pLncfWrna/1fgAA5Kypz6kys1Ml7aggVN0kaQ9Jv5fUtaEKAKD6puqlrRsXlGo93BcAgC6WdqGK/SXtLOkv7n64pH+QNLZlvQIAtF+WUSP30gsAgD6TNlS9Ey6tvjx8VtVLyrBIhZkdYGbzzGzIzAoV504ys/lm9oSZ7RYp3z0sm29mJ0bKNzSze8zsSTO7xszGNNovAGiJ4sp3y5fXrvvuu9KKFc15v3pGhs48M9t7plEZuAhgAIAekTZUDZjZeEk/kjQo6X5J92Z430cl7SfprmihmW0q6WBJm0naXdL3zWykmY2UdKGCaYebSjokrCtJZ0k6z92nSXpd0mcy9AsAWmf0aOnUU5PPm0ljxkij0i7MGlGcXtfodL6ZM+t/z0YQpAAAPSjt6n9Hu/sb7v4DSbtKOiycBtgQd3/M3Z+IOTVD0tXuvtTd/0/SfElbh6/57v60uy+TdLWkGWZmknaS9LPw+ssl7dNovwCg6SpDzte/Hh+aKuvVe//RiIRf5514H9O77xKuAAA9JVWoMrMrzOwIM9vE3Z9x94db1J/1JT0XOV4QliWVv0fSG+6+vKIcADrXihXpHoqbNhBV1qsMLGecMfyaPfZIrt9qjYzEAQDQwdJO/7tM0rqSLjCzp8zsOjM7rtoFZna7mT0a85pR7bKYMm+gPKlPR5rZgJkNvPzyy9W6D6AfnXhi/VPo0kq7iENlnXr7Urw+2s5Xvzq83i231NcuAABIlHb6352SviXp/0m6WFJB0udrXLOLu38g5vWLKpctkLRB5HiypBeqlL8iabyZjaooT+rTLHcvuHth0qRJ1boPoB+ddVZpf7vtsrdX69lNUaecEh+Iiu2kGc2qFtpWXTX5HAAAyCTtc6rukLSKpD9K+p2krdy9FY+0nyPpp2Z2rqT1JE1TsCCGSZpmZhtKel7BYhb/4u5uZr9WsOT71ZIOk1QttAFAOnff3bq2045YVbvPKs1zn6J1Fi9ObgcAAGSSdvrfw5KWSfqApM0lfcDMVm70Tc1sXzNbIGk7STea2VxJcvd5kmZL+pOkWyQd4+4rwnumjpU0V9JjkmaHdSVppqQvm9l8BfdYXdJovwCgTLXQctxxwfmkEa1TTintNxpcql1X6z6qotNPL7+mExeuAACgy5nX8X/2ZraqpMMlHS9pHXfv2gcAFwoFHxgYyLsbADpFUthI+h1ZWf/kk6VvfjP+fLNGg3beWbrzzuHltdpP+tl23126+ebs/QIAoEeZ2aC7F2rVS7v637Fmdo2kBxUsWX6pgmdGAUDvOfHE0n7akZ1vfav1I0F33JF+wYuopGsIVAAANEXadW1XlnSupMHI8uUA0J3GjpWWLo0/N3NmsAT5mWcmX18rOLXjnqVG2i1eMzQkLVnS3P4AANDH0q7+d46k0ZI+KUlmNilcNAIAuouZtGxZefCJ7hfDVDS0VAtRjYwc5W3ECGncuLx7AQBAz0g7/e9UBQtCnBQWjZZ0Zas6BQAtUW01vUZ85COl/WK4cpf+8z+DskWLsrUPAAC6QtrV//aV9AlJiyXJ3V+QtFqrOgUATZcUoGbOTL4mbrQq2s7//E/8dd/4RnDtKqvU10cAANCV0oaqZR4sE+iSZGb8pQAgX2+/Xf7spWpOOKH8OBqWzj47vhwAACCltKFqtpn9UNJ4MztC0u2SLm5dtwCghlVWkVZdNd0UvnPOKe0Xg9PWW6d7n6R7q6JT/wAAQF9LtfosVdQuAAAXUklEQVSfu3/bzHaV9Jak90s6xd1va2nPAKAZvvKV+P177sl2T1XS1D8AANB36nr4798uMhsp6WB3/0nzu9QePPwX6GJxYSjNQ3rj6hTPL1qUfA/UV78aLLOe5v0AAEDPaMrDf81sdTM7ycy+Z2bTLXCspKclHdiszgJAZo2OOg0N1V5U4vTTy4+Z+gcAACJq3VP13wqm+z0i6bOSbpV0gKQZ7j6jxX0DgOrqGS1KM5JVzXbblfaZ+gcAACJq3VO1kbt/UJLM7GJJr0ia4u5/bXnPACBOZQhyL1/uPO1De+v1hz9Ib74prbFG89oEAAA9odZI1bvFHXdfIen/CFQA+haBCgAAxKg1UvUPZvZWuG+SVg6PTZK7++ot7R0AJImOSFUbrQIAAGixqqHK3Ue2qyMAUFM90/miy6cTsgAAQAulek4VAHS8ytEqAACANiFUAeh8cYtTAAAAdIhaC1UAQH7M6ht1igtbBDAAANBihCoA3YWQBAAAOgzT/wB0pugIVb0P+eWeKgAA0EaMVAHoPautFmyPOSbffgAAgL7ASBWAznbccfVf89ZbtesAAAA0CSNVADpPdPref/1Xfv0AAABIgVAFIH8sPgEAALoYoQpAfjbfPBiVGjEifppfI1P/AAAA2oxQBSCdoaEgAB10UHPaM5MeeaR0fP75QYhi6h8AAOgyhCoA6YwcGWxnz87eVtKS5+efn71tAACANiNUAWivykDlHkwDrMTUPwAA0CUIVQDyU1yg4qGHhp9j6h8AAOgShCoA9UuavlfPdZUr/rECIAAA6FI8/BdA5yBYAQCALsRIFYDajjqque0de2xz2wMAAMgRoQpAbT/8YfY2olP/Lrgge3sAAAAdglAFoDGN3lcFAADQYwhVAJrv8MPLQ1e1BSoAAAC6XC6hyswOMLN5ZjZkZoVI+a5mNmhmj4TbnSLnPhyWzzez882Cv9LMbE0zu83Mngy3E/L4mYC+8LnPpat32WXB1owRLQAA0PPyGql6VNJ+ku6qKH9F0j+5+wclHSbpvyPnLpJ0pKRp4Wv3sPxESXe4+zRJd4THAFrhBz+oPdJULUSxQAUAAOhBuYQqd3/M3Z+IKX/A3V8ID+dJWsnMxprZupJWd/c/urtLukLSPmG9GZIuD/cvj5QDaLVao1CVAYwFKgAAQA/q5Huq/lnSA+6+VNL6khZEzi0IyyRpbXdfKEnhdq229hLodfVM34vWPfroYOteegEAAPSglj3818xul7ROzKmT3f0XNa7dTNJZkqYXi2Kq1f0XmpkdqWAKoaZMmVLv5QDqceGFefcAAACgLVoWqtx9l0auM7PJkq6X9Cl3fyosXiBpcqTaZEnFaYIvmtm67r4wnCb4UpU+zZI0S5IKhQL/bA40yn34CNZmm5X2i6NUAAAAfaCjpv+Z2XhJN0o6yd3/p1geTuv7q5ltG6769ylJxdGuOQoWtVC4rToKBqDJiiv8/elPpTJGqQAAQB/Ja0n1fc1sgaTtJN1oZnPDU8dKep+k/2dmD4av4j1Sn5d0saT5kp6SdHNYfqakXc3sSUm7hscAmi3tPVFjxrS2HwAAAB3GvE9vHi8UCj4wMJB3N4DOV5zml/S7onIaYJ/+TgEAAL3HzAbdvVCrXsvuqQLQA9Ks/BcNUQQqAADQhzrqnioAXa6e5dcBAAB6BKEKAAAAADIgVAEAAABABoQqALUNDeXdAwAAgI5FqAIQL3p/FPdKAQAAJCJUAQAAAEAGhCoAAAAAyIBQBWC46HQ/nj0FAABQFaEKAAAAADIgVAEAAABABoQqAOWY+gcAAFAXQhUAAAAAZECoAvrNihU8dwoAAKCJCFVAvxk1KtjGBauRI0v7TP0DAABIhVAF9JPKIFV5PDTUvr4AAAD0CEIV0K3efLO+EFRryh9TAgEAABpCqAK61fjxwXS9uDBkFrz+7u9Kx1HRqX3FuknnAQAAUNWovDsA9L1ioKknyNSaxlf09NP1ByYCFQAAQF0YqQLy1O4pd9HAFBeeCFQAAAB1I1QBnaLRgJUUjqIvSdp33+rXEqgAAAAawvQ/oJO99po0YUJ54IruF4NQMTwlBbNqgYkwBQAAkAmhCuhk73lPsE0TfFi9DwAAIBdM/wM6ySc/WdpPGp0CAABARyFUAXmZPHl42ZVXJtevXPqcaXsAAAAdgel/QCukCT/PP5/uegAAAHQ0RqqAZkv7DKmoeheSeO976+sTAAAAWoZQBbRDPSNPaUa5nnkmU3cAAADQPIQqoF0aWe68njoAAADIBaEKaKbKUaakMPSv/zq8rLJu3PHLLxOwAAAAOgyhCmi1aAgqhq6f/KSxtiZOzN4fAAAANBWhCmiWQw4p7WcdTdprr2zXAwAAoG1YUh1olquvTj7nXhqlqrZoBVP7AAAAug4jVUCeCFEAAABdj1AFNEOaZdDXW689fQEAAEBbEaqAdnn++bx7AAAAgBbIJVSZ2QFmNs/MhsysEHN+ipktMrPjI2W7m9kTZjbfzE6MlG9oZveY2ZNmdo2ZjWnXzwEAAAAAeY1UPSppP0l3JZw/T9LNxQMzGynpQkl7SNpU0iFmtml4+ixJ57n7NEmvS/pMqzoNxDrooNJ+rXukuIcKAACg5+QSqtz9MXd/Iu6cme0j6WlJ8yLFW0ua7+5Pu/sySVdLmmFmJmknST8L610uaZ/W9Rwdxaz6SnrtMnt2Y9ctXdrcfgAAACAXHXVPlZmtImmmpNMqTq0v6bnI8YKw7D2S3nD35RXl6HXRMDVjRn79aIR78BrDTFUAAIBe0LLnVJnZ7ZLWiTl1srv/IuGy0xRM5Vtk5SMQccMRXqU8qU9HSjpSkqZMmZJUDd1mzpy8exBgah8AAEBfalmocvddGrhsG0n7m9nZksZLGjKzJZIGJW0QqTdZ0guSXpE03sxGhaNVxfKkPs2SNEuSCoUCfwH3ErN8Qk0nTD8EAABArloWqhrh7h8r7pvZ1yQtcvfvmdkoSdPMbENJz0s6WNK/uLub2a8l7a/gPqvDJCWNgqFX5BVk3npLWmMNaWiIMAUAAIC/yWtJ9X3NbIGk7STdaGZzq9UPR6GOlTRX0mOSZrt7cSGLmZK+bGbzFdxjdUnreo62WnXV2otRfOITpf1WB5011gi2IzrqVkQAAADkzLxP7wMpFAo+MDCQdzdQTTQk7b239MtfDi93H36ctt24EaficVw7lXXXXlt68cX07wsAAICuYmaD7j7subqV+Cd3dKbKAPOrXw0vLwaZaKCpNVq1336l/coRp/33T24nrt1ioAIAAEBfI1She6Sd3let3vXXJ9e97rr6+wQAAIC+R6hC56l3Ol9cvfHjh9eJjlJVe8/KsgMPLH+fyvcaGkrXRwAAAPQkQhU6X1ywqlX25pvDz0dHqeqZMnjttdXfi5UAAQAA+hqhCp2lkVGqqBkzytuaMCHYX3PNdNdXjkRVC0xxo1YAAADoOx31nCogUeUqf0luuKG83htvDL8uusBFPSGOAAUAAIAYjFShc6QNOGnCz/Tp6d6z2Fb0eVeSNHFiuusBAADQ9xipQrKsU/GarZ4+zA2fJ500SlWr7OWXuVcKAAAAqTBShd4Wve8pyyp9nRAqAQAA0JEIVegMe+1V2m9FgEl7T1blNQAAAEANTP9DZ7jpprx7EI9gBQAAgBoYqUK8ylGdUeRvAAAAIA6hCumsWNGe92FkCAAAAF2G4YdO1mmr77UKq+wBAACgizFSBQAAAAAZEKowXF4jZOPGte+9AAAAgCZh+h/aL2m63+LF7e0HAAAA0ASMVHWqyuCx/fb59CMq671PZtw/BQAAgJ7DSFW3+MMf2v+ezZr6lyZIMUoFAACALkWoQrmdd25eW0lhqpdXMgQAAEDfYfofyt155/CykSPj61abzhdX7k6gAgAAQM9hpKrTued/H9Ly5cP7ED2u1T+CFAAAAHoYI1Wd6CMfiS9v52IVO+4YX77LLvW1Q6ACAABAjzPv0z96C4WCDwwM5N2NeJXPiWrXc6OqvU+t+6PyerYVAAAA0CJmNujuhVr1mP6HQNYphgQpAAAA9Cmm/2G4tAGJIAUAAAAwUtXRiqElbrGKZi5XnmbqXicsmAEAAAB0IEaquk2txSrqCT7/+I+NByVGqQAAAABJjFR1nloh5w9/yD5ilHWUK+m5VQAAAEAfYqSq28U9UPfjH4+vu+OO2QLV0FBQb/nyuroIAAAA9DJCVTeLBqHo/u9+N7yumfTb35aX7bhjfChLwj1VAAAAwDBM/+tUlUEny/Oq4sIQ90QBAAAATcFIVTeqtkJfUTFIVQaqekamAAAAANTESFUnqTW9rt4wFBeoAAAAADQVI1W9Js0oFgAAAICmySVUmdkBZjbPzIbMrFBxbnMz+2N4/hEzWyks/3B4PN/MzjcLhmHMbE0zu83Mngy3E/L4mToagQoAAABombxGqh6VtJ+ku6KFZjZK0pWSjnL3zSTtKOnd8PRFko6UNC187R6WnyjpDnefJumO8Li/Ja0KCAAAAKDpcglV7v6Yuz8Rc2q6pIfd/aGw3qvuvsLM1pW0urv/0d1d0hWS9gmvmSHp8nD/8kh592pGEGJBCgAAAKAtOu2eqo0luZnNNbP7zeyEsHx9SQsi9RaEZZK0trsvlKRwu1bbegsAAACg77Vs9T8zu13SOjGnTnb3X1Tpz0clbSXpbUl3mNmgpLdi6tY9DGNmRyqYQqgpU6bUe3lr8WBdAAAAoCu1LFS5+y4NXLZA0m/d/RVJMrObJG2p4D6ryZF6kyW9EO6/aGbruvvCcJrgS1X6NEvSLEkqFArMjQMAAACQWadN/5sraXMzGxcuWrGDpD+F0/r+ambbhqv+fUpScbRrjqTDwv3DIuUAAAAA0HJ5Lam+r5ktkLSdpBvNbK4kufvrks6VdJ+kByXd7+43hpd9XtLFkuZLekrSzWH5mZJ2NbMnJe0aHnevbbfNuwcAAAAA6mDepyvEFQoFHxgYyLsbJcV7qvr0fw8AAACg05jZoLsXatXrtOl//Wm77fLuAQAAAIAGEao6wd13590DAAAAAA0iVAEAAABABoQqAAAAAMiAUNVJWKQCAAAA6DqEKgAAAADIgFCVt+JS6gAAAAC6EqEKAAAAADIgVAEAAABABoQqAAAAAMhgVN4d6Hus+AcAAAB0NUaqAAAAACADQhUAAAAAZECoAgAAAIAMCFUAAAAAkAGhCgAAAAAyIFQBAAAAQAaEKgAAAADIgFAFAAAAABkQqgAAAAAgA0IVAAAAAGRAqAIAAACADAhVAAAAAJABoQoAAAAAMjB3z7sPuTCzlyU9m3c/QhMlvZJ3J/oAn3Pr8Rm3Hp9x6/EZtx6fcXvwObcen3Hr5f0Zv9fdJ9Wq1LehqpOY2YC7F/LuR6/jc249PuPW4zNuPT7j1uMzbg8+59bjM269bvmMmf4HAAAAABkQqgAAAAAgA0JVZ5iVdwf6BJ9z6/EZtx6fcevxGbcen3F78Dm3Hp9x63XFZ8w9VQAAAACQASNVAAAAAJABoSpnZra7mT1hZvPN7MS8+9MLzGwDM/u1mT1mZvPM7Liw/Gtm9ryZPRi+9sy7r93MzJ4xs0fCz3IgLFvTzG4zsyfD7YS8+9mtzOz9ke/qg2b2lpl9ie9xdmZ2qZm9ZGaPRspiv7sWOD/8Hf2wmW2ZX8+7R8JnfI6ZPR5+jteb2fiwfKqZvRP5Tv8gv553j4TPOPH3g5mdFH6PnzCz3fLpdXdJ+IyviXy+z5jZg2E53+MGVPmbret+JzP9L0dmNlLS/0raVdICSfdJOsTd/5Rrx7qcma0raV13v9/MVpM0KGkfSQdKWuTu3861gz3CzJ6RVHD3VyJlZ0t6zd3PDP+RYIK7z8yrj70i/F3xvKRtJB0uvseZmNnHJS2SdIW7fyAsi/3uhn+UfkHSngo+/++6+zZ59b1bJHzG0yXd6e7LzewsSQo/46mSflWsh3QSPuOvKeb3g5ltKukqSVtLWk/S7ZI2dvcVbe10l4n7jCvOf0fSm+7+db7HjanyN9un1WW/kxmpytfWkua7+9PuvkzS1ZJm5NynrufuC939/nD/r5Iek7R+vr3qGzMkXR7uX67gFyOy21nSU+7eKQ8s72rufpek1yqKk767MxT8QeXufrek8eEfAagi7jN291vdfXl4eLekyW3vWA9J+B4nmSHpandf6u7/J2m+gr9BUEW1z9jMTME/1l7V1k71mCp/s3Xd72RCVb7Wl/Rc5HiB+OO/qcJ/OfqQpHvComPD4eJLmZqWmUu61cwGzezIsGxtd18oBb8oJa2VW+96y8Eq/z9uvsfNl/Td5fd0a/ybpJsjxxua2QNm9lsz+1heneoRcb8f+B4338ckvejuT0bK+B5nUPE3W9f9TiZU5ctiypiP2SRmtqqk6yR9yd3fknSRpL+TtIWkhZK+k2P3esH27r6lpD0kHRNOk0CTmdkYSZ+QdG1YxPe4vfg93WRmdrKk5ZJ+EhYtlDTF3T8k6cuSfmpmq+fVvy6X9PuB73HzHaLyf+zie5xBzN9siVVjyjriu0yoytcCSRtEjidLeiGnvvQUMxut4D/On7j7zyXJ3V909xXuPiTpR2LqQybu/kK4fUnS9Qo+zxeLw/Dh9qX8etgz9pB0v7u/KPE9bqGk7y6/p5vIzA6TtLekQz28qTuckvZquD8o6SlJG+fXy+5V5fcD3+MmMrNRkvaTdE2xjO9x4+L+ZlMX/k4mVOXrPknTzGzD8F+jD5Y0J+c+db1wnvMlkh5z93Mj5dE5t/tKerTyWqRjZquEN5TKzFaRNF3B5zlH0mFhtcMk/SKfHvaUsn8N5XvcMknf3TmSPhWuOLWtgpvSF+bRwW5nZrtLminpE+7+dqR8UrgYi8xsI0nTJD2dTy+7W5XfD3MkHWxmY81sQwWf8b3t7l8P2UXS4+6+oFjA97gxSX+zqQt/J4/KuwP9LFwB6VhJcyWNlHSpu8/LuVu9YHtJn5T0SHGpU0lflXSImW2hYJj4GUmfy6d7PWFtSdcHvws1StJP3f0WM7tP0mwz+4ykP0s6IMc+dj0zG6dgddDod/VsvsfZmNlVknaUNNHMFkg6VdKZiv/u3qRglan5kt5WsPoiakj4jE+SNFbSbeHvjrvd/ShJH5f0dTNbLmmFpKPcPe0CDH0r4TPeMe73g7vPM7PZkv6kYOrlMaz8V1vcZ+zul2j4fa4S3+NGJf3N1nW/k1lSHQAAAAAyYPofAAAAAGRAqAIAAACADAhVAAAAAJABoQoAAAAAMiBUAQAAAEAGhCoAQE8wsxVm9mDkdWIT255qZjwTDAAQi+dUAQB6xTvuvkXenQAA9B9GqgAAPc3MnjGzs8zs3vD1vrD8vWZ2h5k9HG6nhOVrm9n1ZvZQ+PpI2NRIM/uRmc0zs1vNbOWw/hfN7E9hO1fn9GMCAHJEqAIA9IqVK6b/HRQ595a7by3pe5L+Kyz7nqQr3H1zST+RdH5Yfr6k37r7P0jaUtK8sHyapAvdfTNJb0j657D8REkfCts5qlU/HACgc5m7590HAAAyM7NF7r5qTPkzknZy96fNbLSkv7j7e8zsFUnruvu7YflCd59oZi9LmuzuSyNtTJV0m7tPC49nShrt7t80s1skLZJ0g6Qb3H1Ri39UAECHYaQKANAPPGE/qU6cpZH9FSrdl7yXpAslfVjSoJlxvzIA9BlCFQCgHxwU2f4x3P+DpIPD/UMl/T7cv0PS5yXJzEaa2epJjZrZCEkbuPuvJZ0gabykYaNlAIDexr+mAQB6xcpm9mDk+BZ3Ly6rPtbM7lHwj4mHhGVflHSpmX1F0suSDg/Lj5M0y8w+o2BE6vOSFia850hJV5rZGpJM0nnu/kbTfiIAQFfgnioAQE8L76kquPsrefcFANCbmP4HAAAAABkwUgUAAAAAGTBSBQAAAAAZEKoAAAAAIANCFQAAAABkQKgCAAAAgAwIVQAAAACQAaEKAAAAADL4/7/20PDlXEXDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "env = gym.make('Pendulum-v0')\n",
    "agent = Agent(3, 'pendulum')\n",
    "agent.train()\n",
    "optimiser = optim.Adam(agent.parameters(), lr=1e-3)\n",
    "clip = 0.2\n",
    "total_rewards = []\n",
    "\n",
    "plt.figure(figsize=(14, 6))\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Reward')\n",
    "\n",
    "for _ in range(epochs):\n",
    "    total_reward = 0\n",
    "\n",
    "    b_states, b_actions, b_old_log_probs_action, b_returns, b_gaes = [], [], [], [] ,[]  # Collect outputs over a batch\n",
    "    for _ in range(rollouts):\n",
    "        obs, done = obs_to_tensor(env.reset()), False\n",
    "        # Store states and other info because PPO requires calculating policy and value outputs several times\n",
    "        states, actions, rewards, returns, old_log_probs_action, values, gaes = [], [], [], [], [], [], []\n",
    "        with torch.no_grad():\n",
    "            while not done:\n",
    "                states.append(obs)\n",
    "\n",
    "                # Sample an action from the policy conditioned on the current observation\n",
    "                policy, value = agent(obs)\n",
    "                action = policy.sample()\n",
    "                # Take a step in the enviroment based on the action\n",
    "                obs, reward, done, _ = env.step([action.item()])\n",
    "                obs = obs_to_tensor(obs)\n",
    "                total_reward += reward\n",
    "\n",
    "                # Store outputs that will be used for training\n",
    "                actions.append(action)\n",
    "                rewards.append(reward)\n",
    "                old_log_probs_action.append(policy.log_prob(action))\n",
    "                values.append(value)\n",
    "\n",
    "            ep_return, gae = 0, 0\n",
    "            values.append(torch.zeros(1))\n",
    "            trajectory_length = len(rewards)\n",
    "            for i in reversed(range(trajectory_length)):  # Calculate the return backwards\n",
    "                ep_return = rewards[i] + discount * ep_return\n",
    "                td_error = rewards[i] + discount * values[i + 1].item() - values[i].item()\n",
    "                gae = gae * discount * trace_decay + td_error\n",
    "                returns.insert(0, ep_return)\n",
    "                gaes.insert(0, gae)\n",
    "\n",
    "        # Collect data from one episode in the batch\n",
    "        b_states.append(torch.cat(states))\n",
    "        b_actions.append(torch.cat(actions))\n",
    "        b_old_log_probs_action.append(torch.cat(old_log_probs_action))\n",
    "        b_returns.append(torch.tensor(returns))\n",
    "        b_gaes.append(torch.tensor(gaes))\n",
    "\n",
    "    # Batch all data (no need to keep temporal order with a feedforward policy)\n",
    "    b_states = torch.cat(b_states)\n",
    "    b_actions = torch.cat(b_actions)\n",
    "    b_old_log_probs_action = torch.cat(b_old_log_probs_action)\n",
    "    b_returns = torch.cat(b_returns)\n",
    "    b_gaes = torch.cat(b_gaes)\n",
    "    \n",
    "    for _ in range(10):\n",
    "        optimiser.zero_grad()\n",
    "        policies, values = agent(b_states)\n",
    "        ratios = (policies.log_prob(b_actions) - b_old_log_probs_action).exp()  # Policy ratio\n",
    "        loss = -torch.min(ratios * b_gaes, torch.clamp(ratios, min=1 - clip, max=1 + clip) * b_gaes).mean()  # Surrogate loss\n",
    "        loss += value_loss_weight * (b_returns - values).pow(2).mean()\n",
    "        loss -= entropy_loss_weight * policies.entropy().mean()\n",
    "        loss.backward()\n",
    "        optimiser.step()\n",
    "\n",
    "    total_rewards.append(total_reward / rollouts)\n",
    "    plot()\n",
    "\n",
    "clear_output(wait=True)\n",
    "display('Average final reward: %.2f' % total_rewards[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQsAAAD8CAYAAABgtYFHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAA7ZJREFUeJzt3VFOIkEUQNHqCTvSZehqZRnOmnp+jNGJDHfSaNFwzheEVPK+bqoIRS/rug6Ac37NHgDYB7EAErEAErEAErEAErEAErEAErEAErEAksPsAd74GSl8v2XLYjsLIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIBELIDnMHoB9+v38/P764eVl4iT8lGVd19kzjDHGVQxB8zEUH4nG1Vu2LHYM4b+cCsW5z9g/sSATg/smFlyUoNwusQASsQASsQASsQASsQASsQASsQASsQASsQASseCiXCa7XWIBJGIBJGIBJGJB4jYpYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYgEkYsHFeBrZbRMLIBELIBELIBELzvI0MsYQCyASCyARCyARCyARCyARCyARCyARCyA5zB6AfXk8Hj+9f316mjQJP83OAkjEAkjEgk3+PpZwu5Z1XWfPMMYYVzEEp311mezxeHz/zsIf3+zCsmmxWFCdun0qFLuxKRaOIWRfRUEo7oedBdwPOwvg+4kFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkIgFkBxmD/BmmT0A8G92FkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkAiFkDyB781OFi/lORcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.axis('off')\n",
    "\n",
    "agent.eval()\n",
    "obs, done = obs_to_tensor(env.reset()), False\n",
    "view = plt.imshow(env.render(mode='rgb_array'))\n",
    "while not done:\n",
    "    view.set_data(env.render(mode='rgb_array'))\n",
    "    display(plt.gcf())\n",
    "    clear_output(wait=True)\n",
    "    with torch.no_grad():\n",
    "        obs, _, done, _ = env.step([agent(obs)[0].sample().item()])\n",
    "    obs = obs_to_tensor(obs)\n",
    "\n",
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
